[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260929 14:09:58.683665 4041 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.242.126:37097
I20260929 14:09:58.685021 4041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 14:09:58.686414 4041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:09:58.706856 4048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:09:58.707162 4047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:09:58.707090 4050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:09:58.708048 4041 server_base.cc:1034] running on GCE node
I20260929 14:10:00.010741 4041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 14:10:00.011039 4041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 14:10:00.011250 4041 hybrid_clock.cc:648] HybridClock initialized: now 1790691000011231 us; error 0 us; skew 500 ppm
I20260929 14:10:00.012112 4041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 14:10:00.018431 4041 webserver.cc:460] Webserver started at http://127.3.242.126:34537/ using document root <none> and password file <none>
I20260929 14:10:00.019358 4041 fs_manager.cc:359] Metadata directory not provided
I20260929 14:10:00.019615 4041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:10:00.020066 4041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:10:00.026845 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/master-0-root/instance:
uuid: "42be7bc87f17483a96850995c458fb2f"
format_stamp: "Formatted at 2026-09-29 14:10:00 on dist-test-slave-6gv9"
I20260929 14:10:00.035604 4041 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260929 14:10:00.041565 4056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:00.043008 4041 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 14:10:00.043373 4041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/master-0-root
uuid: "42be7bc87f17483a96850995c458fb2f"
format_stamp: "Formatted at 2026-09-29 14:10:00 on dist-test-slave-6gv9"
I20260929 14:10:00.043926 4041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:10:00.082602 4041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:10:00.084213 4041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 14:10:00.084661 4041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:10:00.168689 4041 rpc_server.cc:307] RPC server started. Bound to: 127.3.242.126:37097
I20260929 14:10:00.168948 4107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.242.126:37097 every 8 connection(s)
I20260929 14:10:00.177578 4108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:00.202728 4108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f: Bootstrap starting.
I20260929 14:10:00.209316 4108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:00.211455 4108 log.cc:824] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f: Log is configured to *not* fsync() on all Append() calls
I20260929 14:10:00.216481 4108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f: No bootstrap required, opened a new log
I20260929 14:10:00.240635 4108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42be7bc87f17483a96850995c458fb2f" member_type: VOTER }
I20260929 14:10:00.241191 4108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:00.241532 4108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42be7bc87f17483a96850995c458fb2f, State: Initialized, Role: FOLLOWER
I20260929 14:10:00.242477 4108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42be7bc87f17483a96850995c458fb2f" member_type: VOTER }
I20260929 14:10:00.242956 4108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:10:00.243299 4108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:10:00.243636 4108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:00.247594 4108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42be7bc87f17483a96850995c458fb2f" member_type: VOTER }
I20260929 14:10:00.248348 4108 leader_election.cc:302] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42be7bc87f17483a96850995c458fb2f; no voters:
I20260929 14:10:00.250684 4108 leader_election.cc:288] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:10:00.251099 4111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:00.253891 4111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [term 1 LEADER]: Becoming Leader. State: Replica: 42be7bc87f17483a96850995c458fb2f, State: Running, Role: LEADER
I20260929 14:10:00.255219 4111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42be7bc87f17483a96850995c458fb2f" member_type: VOTER }
I20260929 14:10:00.256237 4108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:10:00.267517 4112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42be7bc87f17483a96850995c458fb2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42be7bc87f17483a96850995c458fb2f" member_type: VOTER } }
I20260929 14:10:00.267797 4113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42be7bc87f17483a96850995c458fb2f. Latest consensus state: current_term: 1 leader_uuid: "42be7bc87f17483a96850995c458fb2f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42be7bc87f17483a96850995c458fb2f" member_type: VOTER } }
I20260929 14:10:00.268254 4112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [sys.catalog]: This master's current role is: LEADER
I20260929 14:10:00.268460 4113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [sys.catalog]: This master's current role is: LEADER
I20260929 14:10:00.272326 4121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:10:00.284559 4121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:10:00.287035 4041 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260929 14:10:00.303982 4121 catalog_manager.cc:1347] Generated new cluster ID: 2359cc51f6524849ba6c7c05ca47626c
I20260929 14:10:00.304303 4121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:10:00.325203 4121 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:10:00.327152 4121 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:10:00.341619 4121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f: Generated new TSK 0
I20260929 14:10:00.342696 4121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:10:00.354233 4041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:10:00.363590 4130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:00.366530 4131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:00.368955 4133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:10:00.369930 4041 server_base.cc:1034] running on GCE node
I20260929 14:10:00.371338 4041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 14:10:00.371610 4041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 14:10:00.371840 4041 hybrid_clock.cc:648] HybridClock initialized: now 1790691000371823 us; error 0 us; skew 500 ppm
I20260929 14:10:00.372689 4041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 14:10:00.376840 4041 webserver.cc:460] Webserver started at http://127.3.242.65:33657/ using document root <none> and password file <none>
I20260929 14:10:00.377981 4041 fs_manager.cc:359] Metadata directory not provided
I20260929 14:10:00.378233 4041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:10:00.378578 4041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:10:00.380293 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/ts-0-root/instance:
uuid: "55e23cb99a754a72b1144e3e226b1eb7"
format_stamp: "Formatted at 2026-09-29 14:10:00 on dist-test-slave-6gv9"
I20260929 14:10:00.387825 4041 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260929 14:10:00.393507 4138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:00.394518 4041 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 14:10:00.394842 4041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/ts-0-root
uuid: "55e23cb99a754a72b1144e3e226b1eb7"
format_stamp: "Formatted at 2026-09-29 14:10:00 on dist-test-slave-6gv9"
I20260929 14:10:00.395210 4041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:10:00.413583 4041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:10:00.415454 4041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:10:00.418363 4041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:10:00.423936 4041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:10:00.424221 4041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:00.424481 4041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:10:00.424670 4041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:00.535913 4041 rpc_server.cc:307] RPC server started. Bound to: 127.3.242.65:42527
I20260929 14:10:00.535981 4200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.242.65:42527 every 8 connection(s)
I20260929 14:10:00.543329 4041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:10:00.559334 4205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:00.565975 4206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:00.568954 4208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:10:00.571914 4041 server_base.cc:1034] running on GCE node
I20260929 14:10:00.573027 4041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 14:10:00.573351 4041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 14:10:00.573570 4041 hybrid_clock.cc:648] HybridClock initialized: now 1790691000573554 us; error 0 us; skew 500 ppm
I20260929 14:10:00.574391 4041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 14:10:00.575189 4201 heartbeater.cc:347] Connected to a master server at 127.3.242.126:37097
I20260929 14:10:00.575644 4201 heartbeater.cc:464] Registering TS with master...
I20260929 14:10:00.576712 4201 heartbeater.cc:511] Master 127.3.242.126:37097 requested a full tablet report, sending...
I20260929 14:10:00.577853 4041 webserver.cc:460] Webserver started at http://127.3.242.66:45813/ using document root <none> and password file <none>
I20260929 14:10:00.578565 4041 fs_manager.cc:359] Metadata directory not provided
I20260929 14:10:00.578752 4041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:10:00.579036 4041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:10:00.580222 4073 ts_manager.cc:194] Registered new tserver with Master: 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527)
I20260929 14:10:00.580567 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/ts-1-root/instance:
uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
format_stamp: "Formatted at 2026-09-29 14:10:00 on dist-test-slave-6gv9"
I20260929 14:10:00.584786 4073 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55850
I20260929 14:10:00.588150 4041 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260929 14:10:00.593127 4213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:00.593962 4041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 14:10:00.594255 4041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/ts-1-root
uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
format_stamp: "Formatted at 2026-09-29 14:10:00 on dist-test-slave-6gv9"
I20260929 14:10:00.594578 4041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:10:00.615546 4041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:10:00.616909 4041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:10:00.620040 4041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:10:00.623211 4041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:10:00.623409 4041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:00.623755 4041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:10:00.623906 4041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:00.737355 4041 rpc_server.cc:307] RPC server started. Bound to: 127.3.242.66:43039
I20260929 14:10:00.737506 4275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.242.66:43039 every 8 connection(s)
I20260929 14:10:00.756596 4041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:10:00.768814 4279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:00.774681 4282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:00.775777 4280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:10:00.776217 4041 server_base.cc:1034] running on GCE node
I20260929 14:10:00.777833 4041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260929 14:10:00.777978 4276 heartbeater.cc:347] Connected to a master server at 127.3.242.126:37097
W20260929 14:10:00.778112 4041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 14:10:00.778304 4041 hybrid_clock.cc:648] HybridClock initialized: now 1790691000778290 us; error 0 us; skew 500 ppm
I20260929 14:10:00.778329 4276 heartbeater.cc:464] Registering TS with master...
I20260929 14:10:00.779088 4041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 14:10:00.779322 4276 heartbeater.cc:511] Master 127.3.242.126:37097 requested a full tablet report, sending...
I20260929 14:10:00.781653 4073 ts_manager.cc:194] Registered new tserver with Master: ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039)
I20260929 14:10:00.782459 4041 webserver.cc:460] Webserver started at http://127.3.242.67:42369/ using document root <none> and password file <none>
I20260929 14:10:00.783123 4041 fs_manager.cc:359] Metadata directory not provided
I20260929 14:10:00.783371 4041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:10:00.783689 4041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:10:00.783980 4073 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55862
I20260929 14:10:00.785279 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/ts-2-root/instance:
uuid: "b3e5999205bc45d3a093e26888f003f5"
format_stamp: "Formatted at 2026-09-29 14:10:00 on dist-test-slave-6gv9"
I20260929 14:10:00.790356 4041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 14:10:00.793684 4287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:00.794512 4041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:10:00.794816 4041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/ts-2-root
uuid: "b3e5999205bc45d3a093e26888f003f5"
format_stamp: "Formatted at 2026-09-29 14:10:00 on dist-test-slave-6gv9"
I20260929 14:10:00.795161 4041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1790690998520575-4041-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:10:00.819080 4041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:10:00.820788 4041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:10:00.823403 4041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:10:00.827688 4041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:10:00.827875 4041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:00.828208 4041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:10:00.828344 4041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:00.938963 4041 rpc_server.cc:307] RPC server started. Bound to: 127.3.242.67:45637
I20260929 14:10:00.939430 4349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.242.67:45637 every 8 connection(s)
I20260929 14:10:00.962368 4350 heartbeater.cc:347] Connected to a master server at 127.3.242.126:37097
I20260929 14:10:00.962770 4350 heartbeater.cc:464] Registering TS with master...
I20260929 14:10:00.963579 4350 heartbeater.cc:511] Master 127.3.242.126:37097 requested a full tablet report, sending...
I20260929 14:10:00.965610 4073 ts_manager.cc:194] Registered new tserver with Master: b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:00.965821 4041 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.02114552s
I20260929 14:10:00.967736 4073 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:55876
I20260929 14:10:00.969502 4041 test_util.cc:268] Using random seed: -1123817215
I20260929 14:10:01.027225 4073 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55888:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:10:01.031307 4073 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:10:01.138350 4315 tablet_service.cc:1467] Processing CreateTablet for tablet f4f71ae1c4a24a1cb89f70436ac6d39f (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260929 14:10:01.138650 4314 tablet_service.cc:1467] Processing CreateTablet for tablet 34d9a8869342456ebb536c8849642565 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260929 14:10:01.140043 4315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f71ae1c4a24a1cb89f70436ac6d39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.141148 4314 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d9a8869342456ebb536c8849642565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.148867 4313 tablet_service.cc:1467] Processing CreateTablet for tablet 57f069c685a74d38ae9570dfec76ad95 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260929 14:10:01.153149 4313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f069c685a74d38ae9570dfec76ad95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.153179 4312 tablet_service.cc:1467] Processing CreateTablet for tablet d9b3984be1cf45f696793fe68b648d3f (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260929 14:10:01.159516 4312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9b3984be1cf45f696793fe68b648d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.168264 4311 tablet_service.cc:1467] Processing CreateTablet for tablet 26d46ca3672f4153b12484ab0eecbfe7 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260929 14:10:01.170532 4311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d46ca3672f4153b12484ab0eecbfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.185302 4310 tablet_service.cc:1467] Processing CreateTablet for tablet 0a3a6ae8c5f44410af8b79ac8a1de584 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260929 14:10:01.188930 4310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a3a6ae8c5f44410af8b79ac8a1de584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.199282 4309 tablet_service.cc:1467] Processing CreateTablet for tablet 5b0afbf24ceb462cbc079ccfbec60c13 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260929 14:10:01.202306 4309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b0afbf24ceb462cbc079ccfbec60c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.212546 4308 tablet_service.cc:1467] Processing CreateTablet for tablet e52a714cc5d542fdadb1fb29c39392d1 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260929 14:10:01.224485 4308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52a714cc5d542fdadb1fb29c39392d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.224900 4165 tablet_service.cc:1467] Processing CreateTablet for tablet 34d9a8869342456ebb536c8849642565 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260929 14:10:01.228780 4166 tablet_service.cc:1467] Processing CreateTablet for tablet f4f71ae1c4a24a1cb89f70436ac6d39f (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260929 14:10:01.233815 4163 tablet_service.cc:1467] Processing CreateTablet for tablet d9b3984be1cf45f696793fe68b648d3f (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260929 14:10:01.235165 4165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d9a8869342456ebb536c8849642565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.228277 4164 tablet_service.cc:1467] Processing CreateTablet for tablet 57f069c685a74d38ae9570dfec76ad95 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260929 14:10:01.247994 4166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f71ae1c4a24a1cb89f70436ac6d39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.238173 4162 tablet_service.cc:1467] Processing CreateTablet for tablet 26d46ca3672f4153b12484ab0eecbfe7 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260929 14:10:01.250113 4161 tablet_service.cc:1467] Processing CreateTablet for tablet 0a3a6ae8c5f44410af8b79ac8a1de584 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260929 14:10:01.253932 4158 tablet_service.cc:1467] Processing CreateTablet for tablet 118c8454885a48248a988813bdba96cc (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260929 14:10:01.253893 4159 tablet_service.cc:1467] Processing CreateTablet for tablet e52a714cc5d542fdadb1fb29c39392d1 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260929 14:10:01.261615 4161 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a3a6ae8c5f44410af8b79ac8a1de584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.264667 4307 tablet_service.cc:1467] Processing CreateTablet for tablet 118c8454885a48248a988813bdba96cc (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260929 14:10:01.266352 4160 tablet_service.cc:1467] Processing CreateTablet for tablet 5b0afbf24ceb462cbc079ccfbec60c13 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260929 14:10:01.271715 4307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118c8454885a48248a988813bdba96cc. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 14:10:01.286667 4120 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260929 14:10:01.289034 4160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b0afbf24ceb462cbc079ccfbec60c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.307451 4158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118c8454885a48248a988813bdba96cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.311249 4241 tablet_service.cc:1467] Processing CreateTablet for tablet f4f71ae1c4a24a1cb89f70436ac6d39f (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260929 14:10:01.318320 4159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52a714cc5d542fdadb1fb29c39392d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.320094 4162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d46ca3672f4153b12484ab0eecbfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.314257 4239 tablet_service.cc:1467] Processing CreateTablet for tablet 57f069c685a74d38ae9570dfec76ad95 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260929 14:10:01.331918 4238 tablet_service.cc:1467] Processing CreateTablet for tablet d9b3984be1cf45f696793fe68b648d3f (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260929 14:10:01.332782 4164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f069c685a74d38ae9570dfec76ad95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.345504 4235 tablet_service.cc:1467] Processing CreateTablet for tablet 5b0afbf24ceb462cbc079ccfbec60c13 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260929 14:10:01.315824 4240 tablet_service.cc:1467] Processing CreateTablet for tablet 34d9a8869342456ebb536c8849642565 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260929 14:10:01.355837 4236 tablet_service.cc:1467] Processing CreateTablet for tablet 0a3a6ae8c5f44410af8b79ac8a1de584 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260929 14:10:01.358261 4241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f71ae1c4a24a1cb89f70436ac6d39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.359869 4163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9b3984be1cf45f696793fe68b648d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.372241 4234 tablet_service.cc:1467] Processing CreateTablet for tablet e52a714cc5d542fdadb1fb29c39392d1 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260929 14:10:01.380646 4240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34d9a8869342456ebb536c8849642565. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.369563 4237 tablet_service.cc:1467] Processing CreateTablet for tablet 26d46ca3672f4153b12484ab0eecbfe7 (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260929 14:10:01.382470 4233 tablet_service.cc:1467] Processing CreateTablet for tablet 118c8454885a48248a988813bdba96cc (DEFAULT_TABLE table=test-workload [id=b4f05a29a4ed45f78589257f70635c41]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260929 14:10:01.388893 4368 tablet_bootstrap.cc:492] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5: Bootstrap starting.
I20260929 14:10:01.400144 4236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a3a6ae8c5f44410af8b79ac8a1de584. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.406145 4235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b0afbf24ceb462cbc079ccfbec60c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.409207 4239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57f069c685a74d38ae9570dfec76ad95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.417517 4238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9b3984be1cf45f696793fe68b648d3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.424079 4237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d46ca3672f4153b12484ab0eecbfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.441037 4234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e52a714cc5d542fdadb1fb29c39392d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.458832 4233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 118c8454885a48248a988813bdba96cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:01.472152 4368 tablet_bootstrap.cc:654] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:01.488245 4369 tablet_bootstrap.cc:492] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7: Bootstrap starting.
I20260929 14:10:01.511490 4370 tablet_bootstrap.cc:492] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db: Bootstrap starting.
I20260929 14:10:01.526854 4368 tablet_bootstrap.cc:492] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5: No bootstrap required, opened a new log
I20260929 14:10:01.527335 4368 ts_tablet_manager.cc:1397] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5: Time spent bootstrapping tablet: real 0.150s user 0.009s sys 0.015s
I20260929 14:10:01.534809 4369 tablet_bootstrap.cc:654] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:01.537851 4370 tablet_bootstrap.cc:654] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:01.552371 4368 raft_consensus.cc:348] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:01.556341 4368 raft_consensus.cc:374] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:01.556636 4368 raft_consensus.cc:729] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Initialized, Role: FOLLOWER
I20260929 14:10:01.560148 4370 tablet_bootstrap.cc:492] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db: No bootstrap required, opened a new log
I20260929 14:10:01.560818 4370 ts_tablet_manager.cc:1397] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.009s
I20260929 14:10:01.557325 4368 consensus_queue.cc:260] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:01.571394 4369 tablet_bootstrap.cc:492] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7: No bootstrap required, opened a new log
I20260929 14:10:01.571832 4369 ts_tablet_manager.cc:1397] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7: Time spent bootstrapping tablet: real 0.088s user 0.007s sys 0.017s
I20260929 14:10:01.573971 4369 raft_consensus.cc:348] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.574594 4369 raft_consensus.cc:374] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:01.574887 4369 raft_consensus.cc:729] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Initialized, Role: FOLLOWER
I20260929 14:10:01.575500 4369 consensus_queue.cc:260] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.577672 4350 heartbeater.cc:503] Master 127.3.242.126:37097 was elected leader, sending a full tablet report...
I20260929 14:10:01.578595 4368 ts_tablet_manager.cc:1428] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5: Time spent starting tablet: real 0.051s user 0.009s sys 0.006s
I20260929 14:10:01.579577 4368 tablet_bootstrap.cc:492] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5: Bootstrap starting.
I20260929 14:10:01.573697 4370 raft_consensus.cc:348] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:01.579888 4370 raft_consensus.cc:374] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:01.580178 4370 raft_consensus.cc:729] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Initialized, Role: FOLLOWER
I20260929 14:10:01.580828 4370 consensus_queue.cc:260] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:01.586467 4368 tablet_bootstrap.cc:654] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:01.590358 4201 heartbeater.cc:503] Master 127.3.242.126:37097 was elected leader, sending a full tablet report...
I20260929 14:10:01.603192 4276 heartbeater.cc:503] Master 127.3.242.126:37097 was elected leader, sending a full tablet report...
I20260929 14:10:01.606698 4370 ts_tablet_manager.cc:1428] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent starting tablet: real 0.045s user 0.014s sys 0.000s
I20260929 14:10:01.607673 4370 tablet_bootstrap.cc:492] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Bootstrap starting.
I20260929 14:10:01.612066 4368 tablet_bootstrap.cc:492] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5: No bootstrap required, opened a new log
I20260929 14:10:01.612591 4368 ts_tablet_manager.cc:1397] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.005s
I20260929 14:10:01.614816 4370 tablet_bootstrap.cc:654] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:01.615638 4368 raft_consensus.cc:348] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.618921 4368 raft_consensus.cc:374] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:01.619239 4368 raft_consensus.cc:729] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Initialized, Role: FOLLOWER
I20260929 14:10:01.620016 4368 consensus_queue.cc:260] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.622251 4368 ts_tablet_manager.cc:1428] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s
I20260929 14:10:01.623107 4368 tablet_bootstrap.cc:492] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5: Bootstrap starting.
I20260929 14:10:01.627802 4369 ts_tablet_manager.cc:1428] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7: Time spent starting tablet: real 0.056s user 0.029s sys 0.015s
I20260929 14:10:01.628969 4369 tablet_bootstrap.cc:492] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7: Bootstrap starting.
I20260929 14:10:01.640743 4368 tablet_bootstrap.cc:654] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:01.643643 4369 tablet_bootstrap.cc:654] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:01.651934 4374 raft_consensus.cc:482] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:01.652456 4374 raft_consensus.cc:504] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.655906 4368 tablet_bootstrap.cc:492] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5: No bootstrap required, opened a new log
I20260929 14:10:01.656433 4368 ts_tablet_manager.cc:1397] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.007s
I20260929 14:10:01.658886 4374 leader_election.cc:288] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039)
I20260929 14:10:01.665359 4368 raft_consensus.cc:348] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.666164 4368 raft_consensus.cc:374] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:01.666460 4368 raft_consensus.cc:729] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Initialized, Role: FOLLOWER
I20260929 14:10:01.666905 4369 tablet_bootstrap.cc:492] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7: No bootstrap required, opened a new log
I20260929 14:10:01.667573 4369 ts_tablet_manager.cc:1397] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7: Time spent bootstrapping tablet: real 0.039s user 0.011s sys 0.013s
I20260929 14:10:01.667176 4368 consensus_queue.cc:260] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.669839 4368 ts_tablet_manager.cc:1428] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260929 14:10:01.671657 4369 raft_consensus.cc:348] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.672613 4369 raft_consensus.cc:374] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:01.672976 4369 raft_consensus.cc:729] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Initialized, Role: FOLLOWER
I20260929 14:10:01.673745 4369 consensus_queue.cc:260] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.675444 4369 ts_tablet_manager.cc:1428] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:10:01.676378 4369 tablet_bootstrap.cc:492] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7: Bootstrap starting.
I20260929 14:10:01.679960 4368 tablet_bootstrap.cc:492] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5: Bootstrap starting.
I20260929 14:10:01.683159 4369 tablet_bootstrap.cc:654] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:01.686610 4368 tablet_bootstrap.cc:654] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:01.689548 4370 tablet_bootstrap.cc:492] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db: No bootstrap required, opened a new log
I20260929 14:10:01.690075 4370 ts_tablet_manager.cc:1397] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent bootstrapping tablet: real 0.083s user 0.020s sys 0.030s
I20260929 14:10:01.692741 4370 raft_consensus.cc:348] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.693503 4370 raft_consensus.cc:374] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:01.694197 4370 raft_consensus.cc:729] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Initialized, Role: FOLLOWER
I20260929 14:10:01.694825 4370 consensus_queue.cc:260] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.695694 4369 tablet_bootstrap.cc:492] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7: No bootstrap required, opened a new log
I20260929 14:10:01.696311 4369 ts_tablet_manager.cc:1397] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.001s
W20260929 14:10:01.697319 4351 tablet.cc:2369] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:10:01.697732 4370 ts_tablet_manager.cc:1428] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20260929 14:10:01.699102 4370 tablet_bootstrap.cc:492] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Bootstrap starting.
I20260929 14:10:01.708364 4369 raft_consensus.cc:348] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.709683 4369 raft_consensus.cc:374] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:01.709954 4369 raft_consensus.cc:729] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Initialized, Role: FOLLOWER
I20260929 14:10:01.710865 4369 consensus_queue.cc:260] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.711705 4251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b3984be1cf45f696793fe68b648d3f" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" is_pre_election: true
I20260929 14:10:01.713491 4369 ts_tablet_manager.cc:1428] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260929 14:10:01.717813 4368 tablet_bootstrap.cc:492] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5: No bootstrap required, opened a new log
I20260929 14:10:01.718451 4368 ts_tablet_manager.cc:1397] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5: Time spent bootstrapping tablet: real 0.048s user 0.013s sys 0.009s
I20260929 14:10:01.716002 4369 tablet_bootstrap.cc:492] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7: Bootstrap starting.
I20260929 14:10:01.720014 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b3984be1cf45f696793fe68b648d3f" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7" is_pre_election: true
W20260929 14:10:01.713549 4290 leader_election.cc:341] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:01.721096 4370 tablet_bootstrap.cc:654] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Neither blocks nor log segments found. Creating new log.
W20260929 14:10:01.722599 4288 leader_election.cc:341] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:01.723028 4288 leader_election.cc:302] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5; no voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db
I20260929 14:10:01.723022 4368 raft_consensus.cc:348] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.723855 4368 raft_consensus.cc:374] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:01.723995 4374 raft_consensus.cc:2726] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:10:01.724164 4368 raft_consensus.cc:729] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Initialized, Role: FOLLOWER
I20260929 14:10:01.725315 4368 consensus_queue.cc:260] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.727550 4368 ts_tablet_manager.cc:1428] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260929 14:10:01.728436 4368 tablet_bootstrap.cc:492] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5: Bootstrap starting.
I20260929 14:10:01.740198 4368 tablet_bootstrap.cc:654] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:01.740309 4369 tablet_bootstrap.cc:654] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:01.744632 4370 tablet_bootstrap.cc:492] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db: No bootstrap required, opened a new log
I20260929 14:10:01.745172 4370 ts_tablet_manager.cc:1397] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent bootstrapping tablet: real 0.046s user 0.007s sys 0.005s
I20260929 14:10:01.747800 4370 raft_consensus.cc:348] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.748698 4370 raft_consensus.cc:374] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:01.749049 4370 raft_consensus.cc:729] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Initialized, Role: FOLLOWER
I20260929 14:10:01.749591 4374 raft_consensus.cc:482] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:01.750193 4374 raft_consensus.cc:504] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.750041 4370 consensus_queue.cc:260] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.752774 4374 leader_election.cc:288] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039)
I20260929 14:10:01.755326 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7" is_pre_election: true
I20260929 14:10:01.755609 4251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" is_pre_election: true
I20260929 14:10:01.756548 4251 raft_consensus.cc:2454] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 0.
W20260929 14:10:01.757416 4288 leader_election.cc:341] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:01.758333 4370 ts_tablet_manager.cc:1428] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260929 14:10:01.758250 4290 leader_election.cc:302] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters: 55e23cb99a754a72b1144e3e226b1eb7
I20260929 14:10:01.759380 4374 raft_consensus.cc:2781] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:01.759457 4370 tablet_bootstrap.cc:492] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db: Bootstrap starting.
I20260929 14:10:01.760296 4374 raft_consensus.cc:482] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:01.760742 4374 raft_consensus.cc:3037] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:01.765252 4368 tablet_bootstrap.cc:492] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5: No bootstrap required, opened a new log
I20260929 14:10:01.765688 4368 ts_tablet_manager.cc:1397] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.015s
I20260929 14:10:01.767194 4370 tablet_bootstrap.cc:654] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:01.767235 4374 raft_consensus.cc:504] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.770098 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:01.770319 4251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:01.769333 4374 leader_election.cc:288] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039)
I20260929 14:10:01.770905 4251 raft_consensus.cc:3037] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:10:01.773557 4288 leader_election.cc:341] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:01.769022 4368 raft_consensus.cc:348] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.774578 4368 raft_consensus.cc:374] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:01.776530 4368 raft_consensus.cc:729] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Initialized, Role: FOLLOWER
I20260929 14:10:01.777691 4368 consensus_queue.cc:260] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.779861 4368 ts_tablet_manager.cc:1428] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260929 14:10:01.780433 4251 raft_consensus.cc:2454] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 1.
I20260929 14:10:01.780920 4368 tablet_bootstrap.cc:492] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5: Bootstrap starting.
I20260929 14:10:01.783475 4290 leader_election.cc:302] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters: 55e23cb99a754a72b1144e3e226b1eb7
I20260929 14:10:01.789325 4374 raft_consensus.cc:2781] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:01.789793 4374 raft_consensus.cc:686] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 1 LEADER]: Becoming Leader. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Running, Role: LEADER
I20260929 14:10:01.790900 4369 tablet_bootstrap.cc:492] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7: No bootstrap required, opened a new log
I20260929 14:10:01.790668 4374 consensus_queue.cc:237] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.791328 4369 ts_tablet_manager.cc:1397] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7: Time spent bootstrapping tablet: real 0.076s user 0.014s sys 0.007s
I20260929 14:10:01.798106 4370 tablet_bootstrap.cc:492] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db: No bootstrap required, opened a new log
I20260929 14:10:01.797924 4369 raft_consensus.cc:348] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.798552 4370 ts_tablet_manager.cc:1397] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.006s
I20260929 14:10:01.798681 4369 raft_consensus.cc:374] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:01.798976 4369 raft_consensus.cc:729] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Initialized, Role: FOLLOWER
I20260929 14:10:01.799664 4369 consensus_queue.cc:260] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.801695 4370 raft_consensus.cc:348] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.802652 4370 raft_consensus.cc:374] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:01.802956 4370 raft_consensus.cc:729] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Initialized, Role: FOLLOWER
I20260929 14:10:01.803572 4370 consensus_queue.cc:260] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.805375 4368 tablet_bootstrap.cc:654] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:01.808034 4369 ts_tablet_manager.cc:1428] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20260929 14:10:01.809092 4369 tablet_bootstrap.cc:492] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7: Bootstrap starting.
I20260929 14:10:01.814096 4370 ts_tablet_manager.cc:1428] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260929 14:10:01.815078 4370 tablet_bootstrap.cc:492] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Bootstrap starting.
I20260929 14:10:01.816716 4369 tablet_bootstrap.cc:654] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:01.827715 4370 tablet_bootstrap.cc:654] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:01.830040 4368 tablet_bootstrap.cc:492] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5: No bootstrap required, opened a new log
I20260929 14:10:01.830540 4368 ts_tablet_manager.cc:1397] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.006s
I20260929 14:10:01.833662 4368 raft_consensus.cc:348] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.834520 4369 tablet_bootstrap.cc:492] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7: No bootstrap required, opened a new log
I20260929 14:10:01.833575 4070 catalog_manager.cc:5495] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3e5999205bc45d3a093e26888f003f5 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "b3e5999205bc45d3a093e26888f003f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: HEALTHY } } }
I20260929 14:10:01.835024 4369 ts_tablet_manager.cc:1397] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7: Time spent bootstrapping tablet: real 0.026s user 0.010s sys 0.009s
I20260929 14:10:01.834451 4368 raft_consensus.cc:374] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:01.837069 4376 raft_consensus.cc:482] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:01.837630 4368 raft_consensus.cc:729] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Initialized, Role: FOLLOWER
I20260929 14:10:01.837774 4376 raft_consensus.cc:504] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:01.840168 4369 raft_consensus.cc:348] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.843369 4374 raft_consensus.cc:482] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:01.879951 4369 raft_consensus.cc:374] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:01.880328 4369 raft_consensus.cc:729] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Initialized, Role: FOLLOWER
I20260929 14:10:01.880122 4374 raft_consensus.cc:504] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:01.882241 4374 leader_election.cc:288] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039), 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527)
I20260929 14:10:01.860350 4376 leader_election.cc:288] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637), 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527)
I20260929 14:10:01.838716 4368 consensus_queue.cc:260] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.863763 4370 tablet_bootstrap.cc:492] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db: No bootstrap required, opened a new log
I20260929 14:10:01.883109 4370 ts_tablet_manager.cc:1397] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent bootstrapping tablet: real 0.068s user 0.026s sys 0.001s
I20260929 14:10:01.882932 4251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" is_pre_election: true
I20260929 14:10:01.881094 4369 consensus_queue.cc:260] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.883555 4251 raft_consensus.cc:2454] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 0.
I20260929 14:10:01.883294 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7" is_pre_election: true
I20260929 14:10:01.841271 4375 raft_consensus.cc:482] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:01.885447 4290 leader_election.cc:302] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
W20260929 14:10:01.886008 4288 leader_election.cc:341] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:01.887673 4389 raft_consensus.cc:2781] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:01.888618 4368 ts_tablet_manager.cc:1428] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5: Time spent starting tablet: real 0.058s user 0.007s sys 0.004s
I20260929 14:10:01.888622 4389 raft_consensus.cc:482] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:01.889081 4389 raft_consensus.cc:3037] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:01.889693 4368 tablet_bootstrap.cc:492] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5: Bootstrap starting.
I20260929 14:10:01.893810 4370 raft_consensus.cc:348] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.895184 4370 raft_consensus.cc:374] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:01.895752 4370 raft_consensus.cc:729] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Initialized, Role: FOLLOWER
I20260929 14:10:01.896791 4370 consensus_queue.cc:260] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.899870 4370 ts_tablet_manager.cc:1428] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20260929 14:10:01.899834 4389 raft_consensus.cc:504] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:01.885932 4375 raft_consensus.cc:504] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:01.902384 4370 tablet_bootstrap.cc:492] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Bootstrap starting.
I20260929 14:10:01.907694 4368 tablet_bootstrap.cc:654] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:01.916079 4251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:01.918565 4251 raft_consensus.cc:3037] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:01.938344 4251 raft_consensus.cc:2454] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 1.
I20260929 14:10:01.917302 4389 leader_election.cc:288] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 election: Requested vote from peers ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039), 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527)
I20260929 14:10:01.957907 4290 leader_election.cc:302] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:01.959092 4389 raft_consensus.cc:2781] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:01.959555 4389 raft_consensus.cc:686] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 1 LEADER]: Becoming Leader. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Running, Role: LEADER
I20260929 14:10:01.960314 4389 consensus_queue.cc:237] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:01.972546 4376 raft_consensus.cc:482] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:01.981648 4376 raft_consensus.cc:504] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.008679 4369 ts_tablet_manager.cc:1428] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7: Time spent starting tablet: real 0.173s user 0.025s sys 0.040s
I20260929 14:10:02.011742 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:02.008940 4376 leader_election.cc:288] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.008369 4375 leader_election.cc:288] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.016301 4369 tablet_bootstrap.cc:492] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7: Bootstrap starting.
I20260929 14:10:02.019233 4400 raft_consensus.cc:482] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:02.030270 4368 tablet_bootstrap.cc:492] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5: No bootstrap required, opened a new log
I20260929 14:10:02.036289 4368 ts_tablet_manager.cc:1397] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5: Time spent bootstrapping tablet: real 0.147s user 0.015s sys 0.008s
W20260929 14:10:02.040755 4288 leader_election.cc:341] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:02.019932 4400 raft_consensus.cc:504] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.050987 4375 raft_consensus.cc:482] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:02.051460 4375 raft_consensus.cc:504] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.053743 4375 leader_election.cc:288] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.054903 4370 tablet_bootstrap.cc:654] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:02.057277 4369 tablet_bootstrap.cc:654] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:02.059873 4368 raft_consensus.cc:348] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.065883 4368 raft_consensus.cc:374] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:02.066689 4368 raft_consensus.cc:729] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Initialized, Role: FOLLOWER
I20260929 14:10:02.071147 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7" is_pre_election: true
I20260929 14:10:02.069985 4400 leader_election.cc:288] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.074496 4324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d9a8869342456ebb536c8849642565" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5" is_pre_election: true
I20260929 14:10:02.074429 4175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d9a8869342456ebb536c8849642565" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7" is_pre_election: true
I20260929 14:10:02.076529 4374 raft_consensus.cc:482] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:02.073407 4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5" is_pre_election: true
I20260929 14:10:02.074424 4389 raft_consensus.cc:482] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:02.073302 4368 consensus_queue.cc:260] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.079461 4374 raft_consensus.cc:504] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
W20260929 14:10:02.085315 4214 leader_election.cc:341] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:02.079942 4175 raft_consensus.cc:2454] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 0.
I20260929 14:10:02.089013 4214 leader_election.cc:302] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:02.089303 4374 leader_election.cc:288] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039)
I20260929 14:10:02.087078 4324 raft_consensus.cc:2454] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 0.
I20260929 14:10:02.090814 4376 raft_consensus.cc:2781] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:02.096187 4376 raft_consensus.cc:482] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:02.096557 4376 raft_consensus.cc:3037] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.096465 4251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" is_pre_election: true
I20260929 14:10:02.094226 4374 raft_consensus.cc:482] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:02.101341 4374 raft_consensus.cc:504] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.086663 4389 raft_consensus.cc:504] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.115013 4389 leader_election.cc:288] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039)
I20260929 14:10:02.103048 4215 leader_election.cc:302] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters: 55e23cb99a754a72b1144e3e226b1eb7, b3e5999205bc45d3a093e26888f003f5
I20260929 14:10:02.115460 4368 ts_tablet_manager.cc:1428] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5: Time spent starting tablet: real 0.074s user 0.007s sys 0.007s
I20260929 14:10:02.116433 4368 tablet_bootstrap.cc:492] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5: Bootstrap starting.
W20260929 14:10:02.104810 4290 leader_election.cc:341] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:02.105691 4374 leader_election.cc:288] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039)
I20260929 14:10:02.101310 4369 tablet_bootstrap.cc:492] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7: No bootstrap required, opened a new log
I20260929 14:10:02.117944 4369 ts_tablet_manager.cc:1397] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7: Time spent bootstrapping tablet: real 0.103s user 0.004s sys 0.020s
I20260929 14:10:02.121594 4250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d46ca3672f4153b12484ab0eecbfe7" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" is_pre_election: true
I20260929 14:10:02.122373 4369 raft_consensus.cc:348] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:02.123585 4369 raft_consensus.cc:374] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:02.123886 4369 raft_consensus.cc:729] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Initialized, Role: FOLLOWER
I20260929 14:10:02.124585 4251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d9a8869342456ebb536c8849642565" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" is_pre_election: true
I20260929 14:10:02.111660 4370 tablet_bootstrap.cc:492] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db: No bootstrap required, opened a new log
I20260929 14:10:02.125592 4370 ts_tablet_manager.cc:1397] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent bootstrapping tablet: real 0.223s user 0.027s sys 0.042s
W20260929 14:10:02.126051 4290 leader_election.cc:341] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:02.128434 4370 raft_consensus.cc:348] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.129138 4370 raft_consensus.cc:374] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:02.129465 4370 raft_consensus.cc:729] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Initialized, Role: FOLLOWER
I20260929 14:10:02.124948 4369 consensus_queue.cc:260] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:02.130064 4370 consensus_queue.cc:260] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.136678 4175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7" is_pre_election: true
I20260929 14:10:02.128113 4072 catalog_manager.cc:5495] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3e5999205bc45d3a093e26888f003f5 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "b3e5999205bc45d3a093e26888f003f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:02.141741 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d9a8869342456ebb536c8849642565" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7" is_pre_election: true
I20260929 14:10:02.145673 4400 raft_consensus.cc:482] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:02.145735 4176 raft_consensus.cc:2454] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 0.
I20260929 14:10:02.141048 4174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d46ca3672f4153b12484ab0eecbfe7" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7" is_pre_election: true
I20260929 14:10:02.146943 4400 raft_consensus.cc:504] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
W20260929 14:10:02.146845 4288 leader_election.cc:341] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527): Illegal state: must be running to vote when last-logged opid is not known
W20260929 14:10:02.147773 4215 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:45637, user_credentials={real_user=slave}} blocked reactor thread for 66359.9us
I20260929 14:10:02.150424 4369 ts_tablet_manager.cc:1428] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7: Time spent starting tablet: real 0.032s user 0.009s sys 0.001s
I20260929 14:10:02.148777 4288 leader_election.cc:302] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5; no voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db
I20260929 14:10:02.151284 4369 tablet_bootstrap.cc:492] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7: Bootstrap starting.
I20260929 14:10:02.151170 4389 raft_consensus.cc:482] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:02.151769 4389 raft_consensus.cc:504] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.153663 4400 leader_election.cc:288] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
W20260929 14:10:02.158798 4288 leader_election.cc:341] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:02.159022 4411 raft_consensus.cc:2726] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:10:02.159274 4288 leader_election.cc:302] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5; no voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db
I20260929 14:10:02.160053 4370 ts_tablet_manager.cc:1428] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent starting tablet: real 0.034s user 0.002s sys 0.015s
I20260929 14:10:02.155578 4250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3a6ae8c5f44410af8b79ac8a1de584" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" is_pre_election: true
I20260929 14:10:02.159333 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3a6ae8c5f44410af8b79ac8a1de584" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7" is_pre_election: true
I20260929 14:10:02.160251 4374 raft_consensus.cc:2726] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:10:02.157624 4389 leader_election.cc:288] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039)
I20260929 14:10:02.162010 4370 tablet_bootstrap.cc:492] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Bootstrap starting.
I20260929 14:10:02.163343 4250 raft_consensus.cc:2454] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 0.
I20260929 14:10:02.162643 4376 raft_consensus.cc:504] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.167076 4290 leader_election.cc:302] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:02.168653 4389 raft_consensus.cc:2781] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:02.168819 4376 leader_election.cc:288] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.167511 4251 raft_consensus.cc:2379] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b3e5999205bc45d3a093e26888f003f5 in current term 1: Already voted for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in this term.
I20260929 14:10:02.169059 4389 raft_consensus.cc:482] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:02.164001 4176 raft_consensus.cc:2454] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 0.
I20260929 14:10:02.165468 4288 leader_election.cc:302] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, b3e5999205bc45d3a093e26888f003f5; no voters:
I20260929 14:10:02.172034 4374 raft_consensus.cc:2781] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:02.172443 4374 raft_consensus.cc:482] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:02.171376 4324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d9a8869342456ebb536c8849642565" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:02.173506 4408 raft_consensus.cc:2726] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:10:02.176249 4369 tablet_bootstrap.cc:654] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:02.172813 4374 raft_consensus.cc:3037] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.171679 4389 raft_consensus.cc:3037] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.180900 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d9a8869342456ebb536c8849642565" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:02.181582 4176 raft_consensus.cc:3037] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.188275 4368 tablet_bootstrap.cc:654] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:02.192560 4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3a6ae8c5f44410af8b79ac8a1de584" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5" is_pre_election: true
I20260929 14:10:02.194080 4374 raft_consensus.cc:504] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.200327 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52a714cc5d542fdadb1fb29c39392d1" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5" is_pre_election: true
I20260929 14:10:02.204583 4322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118c8454885a48248a988813bdba96cc" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5" is_pre_election: true
I20260929 14:10:02.207263 4324 raft_consensus.cc:2379] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in current term 1: Already voted for candidate b3e5999205bc45d3a093e26888f003f5 in this term.
I20260929 14:10:02.207904 4251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3a6ae8c5f44410af8b79ac8a1de584" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" is_pre_election: true
I20260929 14:10:02.208516 4251 raft_consensus.cc:2454] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55e23cb99a754a72b1144e3e226b1eb7 in term 0.
I20260929 14:10:02.208559 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5" is_pre_election: true
I20260929 14:10:02.209069 4250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52a714cc5d542fdadb1fb29c39392d1" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" is_pre_election: true
I20260929 14:10:02.202522 4389 raft_consensus.cc:504] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.214042 4141 leader_election.cc:302] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:02.214362 4325 raft_consensus.cc:2379] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 55e23cb99a754a72b1144e3e226b1eb7 in current term 1: Already voted for candidate b3e5999205bc45d3a093e26888f003f5 in this term.
I20260929 14:10:02.214967 4370 tablet_bootstrap.cc:654] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Neither blocks nor log segments found. Creating new log.
W20260929 14:10:02.215049 4141 leader_election.cc:341] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:02.215173 4389 leader_election.cc:288] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039)
I20260929 14:10:02.216763 4400 raft_consensus.cc:2781] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:02.214067 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118c8454885a48248a988813bdba96cc" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" is_pre_election: true
W20260929 14:10:02.217694 4140 leader_election.cc:341] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:02.218432 4400 raft_consensus.cc:482] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:02.220618 4400 raft_consensus.cc:3037] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:10:02.222035 4141 leader_election.cc:341] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:02.222505 4374 leader_election.cc:288] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039)
I20260929 14:10:02.222807 4141 leader_election.cc:302] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7; no voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db
I20260929 14:10:02.221510 4174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3a6ae8c5f44410af8b79ac8a1de584" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:02.217731 4248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" is_pre_election: true
I20260929 14:10:02.226679 4375 raft_consensus.cc:2726] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260929 14:10:02.221314 4140 leader_election.cc:341] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:02.220441 4250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3a6ae8c5f44410af8b79ac8a1de584" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:02.226788 4248 raft_consensus.cc:2379] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 55e23cb99a754a72b1144e3e226b1eb7 in current term 1: Already voted for candidate b3e5999205bc45d3a093e26888f003f5 in this term.
I20260929 14:10:02.227897 4250 raft_consensus.cc:3037] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.227232 4140 leader_election.cc:302] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7; no voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db
I20260929 14:10:02.230477 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d9a8869342456ebb536c8849642565" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:02.233439 4249 raft_consensus.cc:2379] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b3e5999205bc45d3a093e26888f003f5 in current term 1: Already voted for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in this term.
I20260929 14:10:02.233565 4175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d9a8869342456ebb536c8849642565" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:02.234704 4176 raft_consensus.cc:2454] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 1.
I20260929 14:10:02.236147 4214 leader_election.cc:302] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters: b3e5999205bc45d3a093e26888f003f5
I20260929 14:10:02.233911 4369 tablet_bootstrap.cc:492] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7: No bootstrap required, opened a new log
I20260929 14:10:02.237427 4375 raft_consensus.cc:2726] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:10:02.237740 4376 raft_consensus.cc:2781] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:02.238171 4376 raft_consensus.cc:686] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Becoming Leader. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:02.238348 4288 leader_election.cc:302] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5; no voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db
I20260929 14:10:02.239436 4411 raft_consensus.cc:482] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:02.239369 4376 consensus_queue.cc:237] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.239845 4400 raft_consensus.cc:504] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.241199 4374 raft_consensus.cc:2726] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 14:10:02.239939 4411 raft_consensus.cc:504] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.240470 4369 ts_tablet_manager.cc:1397] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7: Time spent bootstrapping tablet: real 0.089s user 0.012s sys 0.009s
I20260929 14:10:02.243260 4140 leader_election.cc:302] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7; no voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db
I20260929 14:10:02.244637 4174 raft_consensus.cc:2379] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b3e5999205bc45d3a093e26888f003f5 in current term 1: Already voted for candidate 55e23cb99a754a72b1144e3e226b1eb7 in this term.
I20260929 14:10:02.244971 4375 raft_consensus.cc:3037] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.245926 4411 leader_election.cc:288] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.248224 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b3984be1cf45f696793fe68b648d3f" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5" is_pre_election: true
I20260929 14:10:02.248849 4323 raft_consensus.cc:2454] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 0.
I20260929 14:10:02.246517 4175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b3984be1cf45f696793fe68b648d3f" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7" is_pre_election: true
I20260929 14:10:02.250087 4215 leader_election.cc:302] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:02.251729 4369 raft_consensus.cc:348] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
W20260929 14:10:02.252445 4214 leader_election.cc:341] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:02.252723 4369 raft_consensus.cc:374] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:02.253132 4369 raft_consensus.cc:729] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Initialized, Role: FOLLOWER
I20260929 14:10:02.253371 4411 raft_consensus.cc:2781] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:02.253731 4411 raft_consensus.cc:482] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:02.253736 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3a6ae8c5f44410af8b79ac8a1de584" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:02.254195 4411 raft_consensus.cc:3037] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.251122 4400 leader_election.cc:288] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 election: Requested vote from peers ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.258651 4370 tablet_bootstrap.cc:492] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db: No bootstrap required, opened a new log
I20260929 14:10:02.258185 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3a6ae8c5f44410af8b79ac8a1de584" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:02.254160 4369 consensus_queue.cc:260] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.282148 4370 ts_tablet_manager.cc:1397] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent bootstrapping tablet: real 0.121s user 0.009s sys 0.032s
I20260929 14:10:02.282225 4323 raft_consensus.cc:2379] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 55e23cb99a754a72b1144e3e226b1eb7 in current term 1: Already voted for candidate b3e5999205bc45d3a093e26888f003f5 in this term.
I20260929 14:10:02.254233 4250 raft_consensus.cc:2454] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 1.
I20260929 14:10:02.283929 4290 leader_election.cc:302] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:02.284951 4374 raft_consensus.cc:2781] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:02.286537 4374 raft_consensus.cc:686] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 1 LEADER]: Becoming Leader. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Running, Role: LEADER
I20260929 14:10:02.279995 4375 raft_consensus.cc:2726] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:10:02.289000 4140 leader_election.cc:302] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7; no voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db
I20260929 14:10:02.289255 4374 consensus_queue.cc:237] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.290479 4400 raft_consensus.cc:2726] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 14:10:02.291635 4411 raft_consensus.cc:504] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.296005 4375 raft_consensus.cc:482] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:02.297806 4175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b3984be1cf45f696793fe68b648d3f" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:02.298473 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b3984be1cf45f696793fe68b648d3f" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:02.302086 4369 ts_tablet_manager.cc:1428] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7: Time spent starting tablet: real 0.059s user 0.016s sys 0.002s
I20260929 14:10:02.302469 4323 raft_consensus.cc:3037] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.295722 4411 leader_election.cc:288] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.304544 4369 tablet_bootstrap.cc:492] T 26d46ca3672f4153b12484ab0eecbfe7 P 55e23cb99a754a72b1144e3e226b1eb7: Bootstrap starting.
I20260929 14:10:02.296483 4375 raft_consensus.cc:504] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:02.303187 4175 raft_consensus.cc:3037] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.309540 4389 consensus_queue.cc:1035] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:02.313118 4370 raft_consensus.cc:348] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.313892 4370 raft_consensus.cc:374] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:02.314178 4370 raft_consensus.cc:729] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Initialized, Role: FOLLOWER
I20260929 14:10:02.319011 4375 leader_election.cc:288] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637), ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039)
I20260929 14:10:02.317096 4370 consensus_queue.cc:260] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.322350 4250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" is_pre_election: true
I20260929 14:10:02.326248 4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5" is_pre_election: true
I20260929 14:10:02.326941 4250 raft_consensus.cc:2379] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 55e23cb99a754a72b1144e3e226b1eb7 in current term 1: Already voted for candidate b3e5999205bc45d3a093e26888f003f5 in this term.
I20260929 14:10:02.327262 4370 ts_tablet_manager.cc:1428] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent starting tablet: real 0.045s user 0.006s sys 0.001s
I20260929 14:10:02.329046 4370 tablet_bootstrap.cc:492] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Bootstrap starting.
I20260929 14:10:02.329298 4141 leader_election.cc:302] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7; no voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db
I20260929 14:10:02.331673 4375 raft_consensus.cc:3037] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.337091 4369 tablet_bootstrap.cc:654] T 26d46ca3672f4153b12484ab0eecbfe7 P 55e23cb99a754a72b1144e3e226b1eb7: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:02.338502 4374 consensus_queue.cc:1035] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260929 14:10:02.341341 4323 raft_consensus.cc:2454] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 1.
I20260929 14:10:02.346149 4370 tablet_bootstrap.cc:654] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:02.344530 4175 raft_consensus.cc:2454] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 1.
I20260929 14:10:02.350271 4215 leader_election.cc:302] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:02.352298 4411 raft_consensus.cc:2781] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:02.352885 4411 raft_consensus.cc:686] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Becoming Leader. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:02.356419 4411 consensus_queue.cc:237] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.357072 4370 tablet_bootstrap.cc:492] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db: No bootstrap required, opened a new log
I20260929 14:10:02.358023 4370 ts_tablet_manager.cc:1397] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.006s
I20260929 14:10:02.357393 4375 raft_consensus.cc:2726] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:10:02.366830 4369 tablet_bootstrap.cc:492] T 26d46ca3672f4153b12484ab0eecbfe7 P 55e23cb99a754a72b1144e3e226b1eb7: No bootstrap required, opened a new log
I20260929 14:10:02.367293 4369 ts_tablet_manager.cc:1397] T 26d46ca3672f4153b12484ab0eecbfe7 P 55e23cb99a754a72b1144e3e226b1eb7: Time spent bootstrapping tablet: real 0.064s user 0.011s sys 0.004s
I20260929 14:10:02.374991 4370 raft_consensus.cc:348] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.376104 4370 raft_consensus.cc:374] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:02.376369 4370 raft_consensus.cc:729] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Initialized, Role: FOLLOWER
I20260929 14:10:02.377472 4370 consensus_queue.cc:260] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
W20260929 14:10:02.379364 4120 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260929 14:10:02.392486 4370 ts_tablet_manager.cc:1428] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20260929 14:10:02.393536 4370 tablet_bootstrap.cc:492] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db: Bootstrap starting.
I20260929 14:10:02.398389 4368 tablet_bootstrap.cc:492] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5: No bootstrap required, opened a new log
I20260929 14:10:02.398866 4368 ts_tablet_manager.cc:1397] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5: Time spent bootstrapping tablet: real 0.283s user 0.017s sys 0.025s
I20260929 14:10:02.401523 4369 raft_consensus.cc:348] T 26d46ca3672f4153b12484ab0eecbfe7 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.399509 4071 catalog_manager.cc:5495] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3e5999205bc45d3a093e26888f003f5 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "b3e5999205bc45d3a093e26888f003f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: HEALTHY } } }
I20260929 14:10:02.402338 4369 raft_consensus.cc:374] T 26d46ca3672f4153b12484ab0eecbfe7 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:02.401029 4072 catalog_manager.cc:5495] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db reported cstate change: term changed from 0 to 1, leader changed from <none> to ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66). New cstate: current_term: 1 leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:02.402668 4369 raft_consensus.cc:729] T 26d46ca3672f4153b12484ab0eecbfe7 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Initialized, Role: FOLLOWER
I20260929 14:10:02.403328 4369 consensus_queue.cc:260] T 26d46ca3672f4153b12484ab0eecbfe7 P 55e23cb99a754a72b1144e3e226b1eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.405457 4369 ts_tablet_manager.cc:1428] T 26d46ca3672f4153b12484ab0eecbfe7 P 55e23cb99a754a72b1144e3e226b1eb7: Time spent starting tablet: real 0.036s user 0.004s sys 0.004s
I20260929 14:10:02.406247 4369 tablet_bootstrap.cc:492] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7: Bootstrap starting.
I20260929 14:10:02.409627 4368 raft_consensus.cc:348] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.410549 4368 raft_consensus.cc:374] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:02.410827 4368 raft_consensus.cc:729] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Initialized, Role: FOLLOWER
I20260929 14:10:02.411415 4368 consensus_queue.cc:260] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.416467 4370 tablet_bootstrap.cc:654] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:02.420047 4368 ts_tablet_manager.cc:1428] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5: Time spent starting tablet: real 0.021s user 0.003s sys 0.003s
I20260929 14:10:02.421816 4368 tablet_bootstrap.cc:492] T e52a714cc5d542fdadb1fb29c39392d1 P b3e5999205bc45d3a093e26888f003f5: Bootstrap starting.
I20260929 14:10:02.428539 4369 tablet_bootstrap.cc:654] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:02.429540 4368 tablet_bootstrap.cc:654] T e52a714cc5d542fdadb1fb29c39392d1 P b3e5999205bc45d3a093e26888f003f5: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:02.458772 4389 consensus_queue.cc:1035] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:10:02.459442 4370 tablet_bootstrap.cc:492] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db: No bootstrap required, opened a new log
I20260929 14:10:02.459930 4370 ts_tablet_manager.cc:1397] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent bootstrapping tablet: real 0.067s user 0.006s sys 0.006s
I20260929 14:10:02.462451 4370 raft_consensus.cc:348] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.463213 4370 raft_consensus.cc:374] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:02.463464 4370 raft_consensus.cc:729] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Initialized, Role: FOLLOWER
I20260929 14:10:02.464323 4370 consensus_queue.cc:260] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.465723 4368 tablet_bootstrap.cc:492] T e52a714cc5d542fdadb1fb29c39392d1 P b3e5999205bc45d3a093e26888f003f5: No bootstrap required, opened a new log
I20260929 14:10:02.466398 4368 ts_tablet_manager.cc:1397] T e52a714cc5d542fdadb1fb29c39392d1 P b3e5999205bc45d3a093e26888f003f5: Time spent bootstrapping tablet: real 0.046s user 0.019s sys 0.021s
I20260929 14:10:02.466704 4411 raft_consensus.cc:482] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:02.467155 4411 raft_consensus.cc:504] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.469005 4411 leader_election.cc:288] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.469131 4374 consensus_queue.cc:1035] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:02.469599 4370 ts_tablet_manager.cc:1428] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20260929 14:10:02.469981 4368 raft_consensus.cc:348] T e52a714cc5d542fdadb1fb29c39392d1 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.470963 4368 raft_consensus.cc:374] T e52a714cc5d542fdadb1fb29c39392d1 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:02.471482 4368 raft_consensus.cc:729] T e52a714cc5d542fdadb1fb29c39392d1 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Initialized, Role: FOLLOWER
I20260929 14:10:02.471658 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5" is_pre_election: true
I20260929 14:10:02.472259 4323 raft_consensus.cc:2454] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 0.
I20260929 14:10:02.472424 4368 consensus_queue.cc:260] T e52a714cc5d542fdadb1fb29c39392d1 P b3e5999205bc45d3a093e26888f003f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.474500 4215 leader_election.cc:302] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:02.474977 4368 ts_tablet_manager.cc:1428] T e52a714cc5d542fdadb1fb29c39392d1 P b3e5999205bc45d3a093e26888f003f5: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260929 14:10:02.476903 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7" is_pre_election: true
I20260929 14:10:02.478622 4411 raft_consensus.cc:2781] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:02.478977 4411 raft_consensus.cc:482] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:02.479453 4411 raft_consensus.cc:3037] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:10:02.478878 4214 leader_election.cc:341] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:02.486129 4411 raft_consensus.cc:504] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.490196 4411 leader_election.cc:288] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.490792 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:02.491305 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:02.491820 4323 raft_consensus.cc:3037] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:10:02.495117 4214 leader_election.cc:341] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:02.483897 4072 catalog_manager.cc:5495] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db reported cstate change: term changed from 0 to 1, leader changed from <none> to ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66). New cstate: current_term: 1 leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:02.521911 4323 raft_consensus.cc:2454] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 1.
I20260929 14:10:02.525192 4215 leader_election.cc:302] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters: 55e23cb99a754a72b1144e3e226b1eb7
I20260929 14:10:02.527176 4411 raft_consensus.cc:2781] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:02.527639 4411 raft_consensus.cc:686] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Becoming Leader. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:02.528409 4411 consensus_queue.cc:237] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.530431 4369 tablet_bootstrap.cc:492] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7: No bootstrap required, opened a new log
I20260929 14:10:02.530895 4369 ts_tablet_manager.cc:1397] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7: Time spent bootstrapping tablet: real 0.125s user 0.027s sys 0.045s
I20260929 14:10:02.533867 4369 raft_consensus.cc:348] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.534610 4369 raft_consensus.cc:374] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:02.534870 4369 raft_consensus.cc:729] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Initialized, Role: FOLLOWER
I20260929 14:10:02.535498 4369 consensus_queue.cc:260] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.537884 4369 ts_tablet_manager.cc:1428] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7: Time spent starting tablet: real 0.007s user 0.006s sys 0.002s
I20260929 14:10:02.547506 4072 catalog_manager.cc:5495] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db reported cstate change: term changed from 0 to 1, leader changed from <none> to ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66). New cstate: current_term: 1 leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:02.554790 4411 raft_consensus.cc:482] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:02.555222 4411 raft_consensus.cc:504] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.557242 4411 leader_election.cc:288] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.558080 4174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d46ca3672f4153b12484ab0eecbfe7" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7" is_pre_election: true
I20260929 14:10:02.558717 4174 raft_consensus.cc:2454] T 26d46ca3672f4153b12484ab0eecbfe7 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 0.
I20260929 14:10:02.558910 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d46ca3672f4153b12484ab0eecbfe7" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5" is_pre_election: true
I20260929 14:10:02.559535 4323 raft_consensus.cc:2454] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 0.
I20260929 14:10:02.559691 4214 leader_election.cc:302] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:02.560739 4411 raft_consensus.cc:2781] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:02.561137 4411 raft_consensus.cc:482] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:02.561520 4411 raft_consensus.cc:3037] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.568684 4411 raft_consensus.cc:504] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.570883 4411 leader_election.cc:288] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.572054 4174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d46ca3672f4153b12484ab0eecbfe7" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:02.572324 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d46ca3672f4153b12484ab0eecbfe7" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:02.573149 4174 raft_consensus.cc:3037] T 26d46ca3672f4153b12484ab0eecbfe7 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.573457 4323 raft_consensus.cc:3037] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.578806 4323 raft_consensus.cc:2454] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 1.
I20260929 14:10:02.580073 4174 raft_consensus.cc:2454] T 26d46ca3672f4153b12484ab0eecbfe7 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 1.
I20260929 14:10:02.580261 4215 leader_election.cc:302] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:02.581125 4411 raft_consensus.cc:2781] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:02.581717 4411 raft_consensus.cc:686] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Becoming Leader. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:02.582544 4411 consensus_queue.cc:237] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.584331 4376 raft_consensus.cc:482] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:02.584789 4376 raft_consensus.cc:504] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.586796 4376 leader_election.cc:288] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.589397 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118c8454885a48248a988813bdba96cc" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5" is_pre_election: true
I20260929 14:10:02.590256 4323 raft_consensus.cc:2454] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 0.
I20260929 14:10:02.591105 4215 leader_election.cc:302] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:02.591753 4376 raft_consensus.cc:2781] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:02.592070 4376 raft_consensus.cc:482] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:02.592363 4376 raft_consensus.cc:3037] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.592211 4072 catalog_manager.cc:5495] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db reported cstate change: term changed from 0 to 1, leader changed from <none> to ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66). New cstate: current_term: 1 leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:02.589406 4174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118c8454885a48248a988813bdba96cc" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7" is_pre_election: true
I20260929 14:10:02.594074 4174 raft_consensus.cc:2454] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 0.
I20260929 14:10:02.600510 4376 raft_consensus.cc:504] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.603147 4376 leader_election.cc:288] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.603698 4174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118c8454885a48248a988813bdba96cc" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:02.603973 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118c8454885a48248a988813bdba96cc" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:02.604558 4174 raft_consensus.cc:3037] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.604635 4323 raft_consensus.cc:3037] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.608863 4323 raft_consensus.cc:2454] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 1.
I20260929 14:10:02.609895 4215 leader_election.cc:302] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:02.610625 4376 raft_consensus.cc:2781] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:02.611105 4376 raft_consensus.cc:686] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Becoming Leader. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:02.612095 4411 raft_consensus.cc:482] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:02.611892 4376 consensus_queue.cc:237] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.612075 4174 raft_consensus.cc:2454] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 1.
I20260929 14:10:02.612826 4411 raft_consensus.cc:504] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.615635 4174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52a714cc5d542fdadb1fb29c39392d1" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7" is_pre_election: true
I20260929 14:10:02.616114 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52a714cc5d542fdadb1fb29c39392d1" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5" is_pre_election: true
I20260929 14:10:02.616518 4174 raft_consensus.cc:2454] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 0.
I20260929 14:10:02.617197 4323 raft_consensus.cc:2454] T e52a714cc5d542fdadb1fb29c39392d1 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 0.
I20260929 14:10:02.617769 4214 leader_election.cc:302] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:02.619618 4410 raft_consensus.cc:2781] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:02.619997 4410 raft_consensus.cc:482] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:02.620332 4410 raft_consensus.cc:3037] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.620363 4411 leader_election.cc:288] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.627782 4410 raft_consensus.cc:504] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.628209 4072 catalog_manager.cc:5495] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db reported cstate change: term changed from 0 to 1, leader changed from <none> to ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66). New cstate: current_term: 1 leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:02.630699 4174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52a714cc5d542fdadb1fb29c39392d1" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:02.631412 4174 raft_consensus.cc:3037] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.631841 4410 leader_election.cc:288] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.631330 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e52a714cc5d542fdadb1fb29c39392d1" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:02.632536 4323 raft_consensus.cc:3037] T e52a714cc5d542fdadb1fb29c39392d1 P b3e5999205bc45d3a093e26888f003f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:02.640138 4323 raft_consensus.cc:2454] T e52a714cc5d542fdadb1fb29c39392d1 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 1.
I20260929 14:10:02.640563 4174 raft_consensus.cc:2454] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 1.
I20260929 14:10:02.641461 4215 leader_election.cc:302] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:02.642333 4410 raft_consensus.cc:2781] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:02.642798 4410 raft_consensus.cc:686] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Becoming Leader. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:02.643924 4410 consensus_queue.cc:237] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.653728 4072 catalog_manager.cc:5495] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db reported cstate change: term changed from 0 to 1, leader changed from <none> to ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66). New cstate: current_term: 1 leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:02.665611 4410 consensus_queue.cc:1035] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:02.681565 4410 consensus_queue.cc:1035] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:10:02.712217 4323 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f"
dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
mode: GRACEFUL
new_leader_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
from {username='slave'} at 127.0.0.1:52948
I20260929 14:10:02.712970 4323 raft_consensus.cc:595] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 1 LEADER]: Received request to transfer leadership to TS 55e23cb99a754a72b1144e3e226b1eb7
I20260929 14:10:02.729678 4250 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "34d9a8869342456ebb536c8849642565"
dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
mode: GRACEFUL
new_leader_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
from {username='slave'} at 127.0.0.1:33152
I20260929 14:10:02.730228 4250 raft_consensus.cc:595] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Received request to transfer leadership to TS 55e23cb99a754a72b1144e3e226b1eb7
I20260929 14:10:02.735783 4323 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95"
dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
mode: GRACEFUL
new_leader_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
from {username='slave'} at 127.0.0.1:52948
I20260929 14:10:02.736317 4323 raft_consensus.cc:595] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 1 LEADER]: Received request to transfer leadership to TS 55e23cb99a754a72b1144e3e226b1eb7
I20260929 14:10:02.741182 4250 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d9b3984be1cf45f696793fe68b648d3f"
dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
mode: GRACEFUL
new_leader_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
from {username='slave'} at 127.0.0.1:33152
I20260929 14:10:02.741932 4250 raft_consensus.cc:595] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Received request to transfer leadership to TS 55e23cb99a754a72b1144e3e226b1eb7
I20260929 14:10:02.750502 4323 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0a3a6ae8c5f44410af8b79ac8a1de584"
dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
mode: GRACEFUL
new_leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
from {username='slave'} at 127.0.0.1:52948
I20260929 14:10:02.751214 4323 raft_consensus.cc:595] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 1 LEADER]: Received request to transfer leadership to TS ff2ac7ef06c44f568ec2ff0fbc58b5db
I20260929 14:10:02.762311 4250 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "118c8454885a48248a988813bdba96cc"
dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
mode: GRACEFUL
new_leader_uuid: "b3e5999205bc45d3a093e26888f003f5"
from {username='slave'} at 127.0.0.1:33152
I20260929 14:10:02.763015 4250 raft_consensus.cc:595] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Received request to transfer leadership to TS b3e5999205bc45d3a093e26888f003f5
I20260929 14:10:02.829934 4431 consensus_queue.cc:1035] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:02.840710 4408 consensus_queue.cc:1035] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:02.846061 4389 raft_consensus.cc:979] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5: : Instructing follower 55e23cb99a754a72b1144e3e226b1eb7 to start an election
I20260929 14:10:02.846518 4389 raft_consensus.cc:1067] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 1 LEADER]: Signalling peer 55e23cb99a754a72b1144e3e226b1eb7 to start an election
I20260929 14:10:02.848968 4176 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95"
dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
from {username='slave'} at 127.0.0.1:34810
I20260929 14:10:02.849586 4176 raft_consensus.cc:482] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:02.849931 4176 raft_consensus.cc:3037] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:02.858820 4176 raft_consensus.cc:504] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.860276 4410 consensus_queue.cc:1035] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:02.862200 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:02.862973 4249 raft_consensus.cc:3037] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:02.868831 4176 leader_election.cc:288] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 2 election: Requested vote from peers ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.869877 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:02.870709 4408 raft_consensus.cc:979] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5: : Instructing follower ff2ac7ef06c44f568ec2ff0fbc58b5db to start an election
I20260929 14:10:02.870980 4323 raft_consensus.cc:3032] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:10:02.871151 4408 raft_consensus.cc:1067] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 1 LEADER]: Signalling peer ff2ac7ef06c44f568ec2ff0fbc58b5db to start an election
I20260929 14:10:02.871343 4323 raft_consensus.cc:729] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Running, Role: LEADER
I20260929 14:10:02.872211 4323 consensus_queue.cc:260] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.873739 4323 raft_consensus.cc:3037] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:02.873736 4250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0a3a6ae8c5f44410af8b79ac8a1de584"
dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
from {username='slave'} at 127.0.0.1:33136
I20260929 14:10:02.874445 4250 raft_consensus.cc:482] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:02.874815 4250 raft_consensus.cc:3037] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:02.873143 4249 raft_consensus.cc:2454] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e23cb99a754a72b1144e3e226b1eb7 in term 2.
I20260929 14:10:02.878716 4141 leader_election.cc:302] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:02.879823 4375 raft_consensus.cc:2781] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:10:02.882781 4250 raft_consensus.cc:504] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.886153 4174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3a6ae8c5f44410af8b79ac8a1de584" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:02.886969 4174 raft_consensus.cc:3037] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:02.887835 4323 raft_consensus.cc:2454] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e23cb99a754a72b1144e3e226b1eb7 in term 2.
I20260929 14:10:02.893769 4174 raft_consensus.cc:2454] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 2.
I20260929 14:10:02.894727 4411 raft_consensus.cc:979] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db: : Instructing follower 55e23cb99a754a72b1144e3e226b1eb7 to start an election
I20260929 14:10:02.895155 4411 raft_consensus.cc:1067] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Signalling peer 55e23cb99a754a72b1144e3e226b1eb7 to start an election
I20260929 14:10:02.896432 4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3a6ae8c5f44410af8b79ac8a1de584" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:02.897073 4325 raft_consensus.cc:3032] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:10:02.897403 4325 raft_consensus.cc:729] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Running, Role: LEADER
I20260929 14:10:02.898135 4325 consensus_queue.cc:260] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.899360 4325 raft_consensus.cc:3037] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:02.901463 4214 leader_election.cc:302] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:02.902459 4411 raft_consensus.cc:2781] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:10:02.902977 4411 raft_consensus.cc:686] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 LEADER]: Becoming Leader. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:02.903749 4176 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9b3984be1cf45f696793fe68b648d3f"
dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
from {username='slave'} at 127.0.0.1:34814
I20260929 14:10:02.903805 4411 consensus_queue.cc:237] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.904845 4176 raft_consensus.cc:482] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:02.905421 4176 raft_consensus.cc:3037] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:02.907123 4410 consensus_queue.cc:1035] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:10:02.909456 4375 raft_consensus.cc:686] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Becoming Leader. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Running, Role: LEADER
I20260929 14:10:02.910262 4375 consensus_queue.cc:237] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.918499 4176 raft_consensus.cc:504] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.921593 4325 raft_consensus.cc:2454] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 2.
I20260929 14:10:02.923015 4324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b3984be1cf45f696793fe68b648d3f" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:02.923638 4324 raft_consensus.cc:3037] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:02.920907 4176 leader_election.cc:288] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 2 election: Requested vote from peers ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.928335 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b3984be1cf45f696793fe68b648d3f" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:02.928994 4249 raft_consensus.cc:3032] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:10:02.929414 4249 raft_consensus.cc:729] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:02.930279 4249 consensus_queue.cc:260] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.932121 4250 leader_election.cc:288] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 2 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:02.931586 4072 catalog_manager.cc:5495] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 reported cstate change: term changed from 1 to 2, leader changed from b3e5999205bc45d3a093e26888f003f5 (127.3.242.67) to 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65). New cstate: current_term: 2 leader_uuid: "55e23cb99a754a72b1144e3e226b1eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:02.934180 4071 catalog_manager.cc:5495] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db reported cstate change: term changed from 1 to 2, leader changed from b3e5999205bc45d3a093e26888f003f5 (127.3.242.67) to ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66). New cstate: current_term: 2 leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:02.937520 4324 raft_consensus.cc:2454] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e23cb99a754a72b1144e3e226b1eb7 in term 2.
I20260929 14:10:02.941586 4249 raft_consensus.cc:3037] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:02.942085 4174 raft_consensus.cc:1226] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Rejecting Update request from peer ff2ac7ef06c44f568ec2ff0fbc58b5db for earlier term 1. Current term is 2. Ops: []
I20260929 14:10:02.943399 4140 leader_election.cc:302] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, b3e5999205bc45d3a093e26888f003f5; no voters:
I20260929 14:10:02.944253 4375 raft_consensus.cc:2781] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:10:02.944818 4375 raft_consensus.cc:686] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Becoming Leader. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Running, Role: LEADER
I20260929 14:10:02.946952 4375 consensus_queue.cc:237] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:02.954873 4249 raft_consensus.cc:2454] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e23cb99a754a72b1144e3e226b1eb7 in term 2.
I20260929 14:10:02.957691 4072 catalog_manager.cc:5495] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 reported cstate change: term changed from 1 to 2, leader changed from ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66) to 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65). New cstate: current_term: 2 leader_uuid: "55e23cb99a754a72b1144e3e226b1eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:02.971315 4410 consensus_queue.cc:1035] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:02.984091 4411 consensus_queue.cc:1035] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:03.010308 4411 consensus_queue.cc:1035] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:03.011833 4374 raft_consensus.cc:979] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5: : Instructing follower 55e23cb99a754a72b1144e3e226b1eb7 to start an election
I20260929 14:10:03.012442 4374 raft_consensus.cc:1067] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 1 LEADER]: Signalling peer 55e23cb99a754a72b1144e3e226b1eb7 to start an election
I20260929 14:10:03.014452 4174 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f"
dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
from {username='slave'} at 127.0.0.1:34810
I20260929 14:10:03.015100 4174 raft_consensus.cc:482] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:03.015410 4174 raft_consensus.cc:3037] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:03.020761 4375 raft_consensus.cc:482] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:03.021718 4375 raft_consensus.cc:504] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:03.022614 4174 raft_consensus.cc:504] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:03.023917 4375 leader_election.cc:288] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:03.024461 4174 leader_election.cc:288] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 2 election: Requested vote from peers b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637), ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039)
I20260929 14:10:03.025679 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" is_pre_election: true
I20260929 14:10:03.025993 4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3e5999205bc45d3a093e26888f003f5" is_pre_election: true
I20260929 14:10:03.026777 4325 raft_consensus.cc:2379] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 55e23cb99a754a72b1144e3e226b1eb7 in current term 1: Already voted for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in this term.
I20260929 14:10:03.027081 4323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:03.027671 4323 raft_consensus.cc:3032] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:10:03.028056 4323 raft_consensus.cc:729] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Running, Role: LEADER
I20260929 14:10:03.029277 4323 consensus_queue.cc:260] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:03.029644 4140 leader_election.cc:302] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7; no voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db
I20260929 14:10:03.027582 4250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:03.030454 4323 raft_consensus.cc:3037] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:03.030833 4250 raft_consensus.cc:3037] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:03.033788 4410 raft_consensus.cc:979] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db: : Instructing follower b3e5999205bc45d3a093e26888f003f5 to start an election
I20260929 14:10:03.034178 4410 raft_consensus.cc:1067] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Signalling peer b3e5999205bc45d3a093e26888f003f5 to start an election
I20260929 14:10:03.035976 4325 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "118c8454885a48248a988813bdba96cc"
dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
from {username='slave'} at 127.0.0.1:52930
I20260929 14:10:03.036533 4325 raft_consensus.cc:482] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:03.037046 4325 raft_consensus.cc:3037] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:03.040275 4375 raft_consensus.cc:3037] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:03.041647 4410 consensus_queue.cc:1035] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:10:03.042482 4250 raft_consensus.cc:2454] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e23cb99a754a72b1144e3e226b1eb7 in term 2.
I20260929 14:10:03.043579 4141 leader_election.cc:302] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:03.044412 4443 raft_consensus.cc:2781] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:10:03.051050 4325 raft_consensus.cc:504] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:03.051601 4323 raft_consensus.cc:2454] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e23cb99a754a72b1144e3e226b1eb7 in term 2.
I20260929 14:10:03.054203 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118c8454885a48248a988813bdba96cc" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:03.054808 4249 raft_consensus.cc:3032] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:10:03.055099 4249 raft_consensus.cc:729] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:03.055058 4175 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "118c8454885a48248a988813bdba96cc" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:03.055979 4249 consensus_queue.cc:260] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:03.051090 4375 raft_consensus.cc:2726] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:10:03.057483 4249 raft_consensus.cc:3037] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:03.062146 4324 raft_consensus.cc:1226] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Rejecting Update request from peer ff2ac7ef06c44f568ec2ff0fbc58b5db for earlier term 1. Current term is 2. Ops: []
I20260929 14:10:03.053129 4325 leader_election.cc:288] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 2 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039)
I20260929 14:10:03.066370 4443 raft_consensus.cc:686] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Becoming Leader. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Running, Role: LEADER
I20260929 14:10:03.067597 4443 consensus_queue.cc:237] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:03.074275 4249 raft_consensus.cc:2454] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 2.
I20260929 14:10:03.075521 4290 leader_election.cc:302] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters: 55e23cb99a754a72b1144e3e226b1eb7
I20260929 14:10:03.076273 4410 consensus_queue.cc:1035] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:03.076753 4374 raft_consensus.cc:2781] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:10:03.077472 4374 raft_consensus.cc:686] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [term 2 LEADER]: Becoming Leader. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Running, Role: LEADER
I20260929 14:10:03.077297 4071 catalog_manager.cc:5495] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 reported cstate change: term changed from 1 to 2, leader changed from b3e5999205bc45d3a093e26888f003f5 (127.3.242.67) to 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65). New cstate: current_term: 2 leader_uuid: "55e23cb99a754a72b1144e3e226b1eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: HEALTHY } } }
I20260929 14:10:03.078279 4374 consensus_queue.cc:237] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:03.099121 4411 consensus_queue.cc:1035] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:03.100353 4071 catalog_manager.cc:5495] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 reported cstate change: term changed from 1 to 2, leader changed from ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66) to b3e5999205bc45d3a093e26888f003f5 (127.3.242.67). New cstate: current_term: 2 leader_uuid: "b3e5999205bc45d3a093e26888f003f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: HEALTHY } } }
I20260929 14:10:03.118834 4411 consensus_queue.cc:1035] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:03.134227 4411 consensus_queue.cc:1035] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:03.186998 4376 raft_consensus.cc:979] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db: : Instructing follower 55e23cb99a754a72b1144e3e226b1eb7 to start an election
I20260929 14:10:03.187484 4411 raft_consensus.cc:1067] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Signalling peer 55e23cb99a754a72b1144e3e226b1eb7 to start an election
I20260929 14:10:03.188817 4176 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34d9a8869342456ebb536c8849642565"
dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
from {username='slave'} at 127.0.0.1:34814
I20260929 14:10:03.189344 4176 raft_consensus.cc:482] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:03.189653 4176 raft_consensus.cc:3037] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:03.194167 4176 raft_consensus.cc:504] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:03.196213 4176 leader_election.cc:288] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 2 election: Requested vote from peers ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:03.196666 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d9a8869342456ebb536c8849642565" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:03.197320 4249 raft_consensus.cc:3032] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:10:03.197635 4249 raft_consensus.cc:729] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:03.199038 4249 consensus_queue.cc:260] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:03.200316 4249 raft_consensus.cc:3037] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:03.200536 4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d9a8869342456ebb536c8849642565" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:03.201077 4325 raft_consensus.cc:3037] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:03.205749 4325 raft_consensus.cc:2454] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e23cb99a754a72b1144e3e226b1eb7 in term 2.
I20260929 14:10:03.206718 4140 leader_election.cc:302] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, b3e5999205bc45d3a093e26888f003f5; no voters:
I20260929 14:10:03.206895 4249 raft_consensus.cc:2454] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e23cb99a754a72b1144e3e226b1eb7 in term 2.
I20260929 14:10:03.207403 4443 raft_consensus.cc:2781] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:10:03.207861 4443 raft_consensus.cc:686] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Becoming Leader. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Running, Role: LEADER
I20260929 14:10:03.208855 4443 consensus_queue.cc:237] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:03.215449 4071 catalog_manager.cc:5495] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 reported cstate change: term changed from 1 to 2, leader changed from ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66) to 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65). New cstate: current_term: 2 leader_uuid: "55e23cb99a754a72b1144e3e226b1eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:03.326196 4325 raft_consensus.cc:1261] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Refusing update from remote peer ff2ac7ef06c44f568ec2ff0fbc58b5db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:03.327693 4411 consensus_queue.cc:1035] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:03.336021 4176 raft_consensus.cc:1261] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Refusing update from remote peer ff2ac7ef06c44f568ec2ff0fbc58b5db: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:03.337569 4376 consensus_queue.cc:1035] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:10:03.366349 4249 raft_consensus.cc:1261] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Refusing update from remote peer 55e23cb99a754a72b1144e3e226b1eb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:03.367573 4443 consensus_queue.cc:1035] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:03.376055 4325 raft_consensus.cc:1261] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Refusing update from remote peer 55e23cb99a754a72b1144e3e226b1eb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:03.377877 4449 consensus_queue.cc:1035] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:03.402235 4249 raft_consensus.cc:1261] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Refusing update from remote peer 55e23cb99a754a72b1144e3e226b1eb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:03.406533 4449 consensus_queue.cc:1035] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:03.419407 4325 raft_consensus.cc:1261] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Refusing update from remote peer 55e23cb99a754a72b1144e3e226b1eb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:03.421106 4375 consensus_queue.cc:1035] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:03.469486 4249 raft_consensus.cc:1261] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Refusing update from remote peer b3e5999205bc45d3a093e26888f003f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:03.471449 4374 consensus_queue.cc:1035] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:10:03.477053 4176 raft_consensus.cc:3037] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:03.484941 4176 raft_consensus.cc:1261] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Refusing update from remote peer b3e5999205bc45d3a093e26888f003f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:03.486747 4408 consensus_queue.cc:1035] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:03.623208 4324 raft_consensus.cc:1261] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Refusing update from remote peer 55e23cb99a754a72b1144e3e226b1eb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:03.624462 4375 consensus_queue.cc:1035] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:03.633337 4249 raft_consensus.cc:1261] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Refusing update from remote peer 55e23cb99a754a72b1144e3e226b1eb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:03.634691 4375 consensus_queue.cc:1035] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:10:03.822979 4324 raft_consensus.cc:1261] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Refusing update from remote peer 55e23cb99a754a72b1144e3e226b1eb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:03.824285 4375 consensus_queue.cc:1035] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:03.832401 4249 raft_consensus.cc:1261] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Refusing update from remote peer 55e23cb99a754a72b1144e3e226b1eb7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:03.833957 4448 consensus_queue.cc:1035] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:05.796564 4041 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260929 14:10:05.797411 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:05.814759 4176 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95"
dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
mode: GRACEFUL
new_leader_uuid: "b3e5999205bc45d3a093e26888f003f5"
from {username='slave'} at 127.0.0.1:34824
I20260929 14:10:05.815281 4176 raft_consensus.cc:595] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Received request to transfer leadership to TS b3e5999205bc45d3a093e26888f003f5
I20260929 14:10:05.827445 4249 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13"
dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
mode: GRACEFUL
new_leader_uuid: "b3e5999205bc45d3a093e26888f003f5"
from {username='slave'} at 127.0.0.1:33162
I20260929 14:10:05.828060 4249 raft_consensus.cc:595] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Received request to transfer leadership to TS b3e5999205bc45d3a093e26888f003f5
I20260929 14:10:05.829125 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260929 14:10:05.829666 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:05.844193 4375 raft_consensus.cc:979] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7: : Instructing follower b3e5999205bc45d3a093e26888f003f5 to start an election
I20260929 14:10:05.844611 4468 raft_consensus.cc:1067] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Signalling peer b3e5999205bc45d3a093e26888f003f5 to start an election
I20260929 14:10:05.845938 4324 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95"
dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
from {username='slave'} at 127.0.0.1:52946
I20260929 14:10:05.846335 4324 raft_consensus.cc:482] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:05.846566 4324 raft_consensus.cc:3037] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:05.850608 4324 raft_consensus.cc:504] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:05.852164 4324 leader_election.cc:288] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 3 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039)
I20260929 14:10:05.853031 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:05.853140 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:05.853754 4249 raft_consensus.cc:3037] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:05.853732 4176 raft_consensus.cc:3032] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:10:05.854489 4176 raft_consensus.cc:729] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Running, Role: LEADER
I20260929 14:10:05.855322 4176 consensus_queue.cc:260] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:05.856637 4176 raft_consensus.cc:3037] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:05.861085 4249 raft_consensus.cc:2454] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 3.
I20260929 14:10:05.862445 4290 leader_election.cc:302] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:05.863293 4463 raft_consensus.cc:2781] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:10:05.863464 4176 raft_consensus.cc:2454] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 3.
I20260929 14:10:05.864605 4463 raft_consensus.cc:686] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 3 LEADER]: Becoming Leader. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Running, Role: LEADER
I20260929 14:10:05.865430 4463 consensus_queue.cc:237] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:05.872541 4070 catalog_manager.cc:5495] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 reported cstate change: term changed from 2 to 3, leader changed from 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65) to b3e5999205bc45d3a093e26888f003f5 (127.3.242.67). New cstate: current_term: 3 leader_uuid: "b3e5999205bc45d3a093e26888f003f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: HEALTHY } } }
I20260929 14:10:06.050488 4464 raft_consensus.cc:979] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db: : Instructing follower b3e5999205bc45d3a093e26888f003f5 to start an election
I20260929 14:10:06.051062 4475 raft_consensus.cc:1067] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Signalling peer b3e5999205bc45d3a093e26888f003f5 to start an election
I20260929 14:10:06.053643 4324 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13"
dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
from {username='slave'} at 127.0.0.1:52930
I20260929 14:10:06.054363 4324 raft_consensus.cc:482] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:06.054697 4324 raft_consensus.cc:3037] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:06.059702 4324 raft_consensus.cc:504] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:06.061476 4324 leader_election.cc:288] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 2 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039)
I20260929 14:10:06.062249 4176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:06.062480 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:06.062922 4176 raft_consensus.cc:3037] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:06.063067 4249 raft_consensus.cc:3032] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:10:06.063462 4249 raft_consensus.cc:729] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:06.064256 4249 consensus_queue.cc:260] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:06.065694 4249 raft_consensus.cc:3037] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:06.070678 4176 raft_consensus.cc:2454] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 2.
I20260929 14:10:06.071895 4288 leader_election.cc:302] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, b3e5999205bc45d3a093e26888f003f5; no voters:
I20260929 14:10:06.072739 4463 raft_consensus.cc:2781] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:10:06.073280 4463 raft_consensus.cc:686] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 2 LEADER]: Becoming Leader. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Running, Role: LEADER
I20260929 14:10:06.073313 4249 raft_consensus.cc:2454] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 2.
I20260929 14:10:06.074379 4463 consensus_queue.cc:237] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:06.083953 4070 catalog_manager.cc:5495] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 reported cstate change: term changed from 1 to 2, leader changed from ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66) to b3e5999205bc45d3a093e26888f003f5 (127.3.242.67). New cstate: current_term: 2 leader_uuid: "b3e5999205bc45d3a093e26888f003f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: HEALTHY } } }
I20260929 14:10:06.268813 4176 raft_consensus.cc:1261] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Refusing update from remote peer b3e5999205bc45d3a093e26888f003f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:06.270020 4463 consensus_queue.cc:1035] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:06.279343 4249 raft_consensus.cc:1261] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Refusing update from remote peer b3e5999205bc45d3a093e26888f003f5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:06.280877 4469 consensus_queue.cc:1035] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:06.679679 4174 raft_consensus.cc:1261] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Refusing update from remote peer b3e5999205bc45d3a093e26888f003f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:06.680946 4469 consensus_queue.cc:1035] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:06.683889 4249 raft_consensus.cc:1261] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Refusing update from remote peer b3e5999205bc45d3a093e26888f003f5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:06.685356 4469 consensus_queue.cc:1035] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:06.830613 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:06.835281 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:06.835677 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:07.836493 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:07.841050 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:07.841447 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:08.842602 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:08.848022 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:08.848435 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:09.849388 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:09.856274 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:09.856796 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:10.857704 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:10.862382 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:10.862818 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:11.863725 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:11.868896 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:11.869269 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:12.870056 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:12.875783 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:12.876124 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:13.876892 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:13.882318 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:13.882798 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:14.883590 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:14.888144 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:14.888595 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:15.917029 4041 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260929 14:10:15.938143 4174 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f"
dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
mode: GRACEFUL
new_leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
from {username='slave'} at 127.0.0.1:50228
I20260929 14:10:15.938791 4174 raft_consensus.cc:595] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Received request to transfer leadership to TS ff2ac7ef06c44f568ec2ff0fbc58b5db
I20260929 14:10:15.943615 4174 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "34d9a8869342456ebb536c8849642565"
dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
mode: GRACEFUL
new_leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
from {username='slave'} at 127.0.0.1:50228
I20260929 14:10:15.944056 4174 raft_consensus.cc:595] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Received request to transfer leadership to TS ff2ac7ef06c44f568ec2ff0fbc58b5db
I20260929 14:10:15.948885 4324 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95"
dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
mode: GRACEFUL
new_leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
from {username='slave'} at 127.0.0.1:52948
I20260929 14:10:15.949462 4324 raft_consensus.cc:595] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 3 LEADER]: Received request to transfer leadership to TS ff2ac7ef06c44f568ec2ff0fbc58b5db
I20260929 14:10:15.954432 4174 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d9b3984be1cf45f696793fe68b648d3f"
dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
mode: GRACEFUL
new_leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
from {username='slave'} at 127.0.0.1:50228
I20260929 14:10:15.954962 4174 raft_consensus.cc:595] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Received request to transfer leadership to TS ff2ac7ef06c44f568ec2ff0fbc58b5db
I20260929 14:10:15.964257 4324 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13"
dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
mode: GRACEFUL
new_leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
from {username='slave'} at 127.0.0.1:52948
I20260929 14:10:15.964733 4324 raft_consensus.cc:595] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 2 LEADER]: Received request to transfer leadership to TS ff2ac7ef06c44f568ec2ff0fbc58b5db
I20260929 14:10:16.077986 4492 raft_consensus.cc:979] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5: : Instructing follower ff2ac7ef06c44f568ec2ff0fbc58b5db to start an election
I20260929 14:10:16.078504 4498 raft_consensus.cc:1067] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 3 LEADER]: Signalling peer ff2ac7ef06c44f568ec2ff0fbc58b5db to start an election
I20260929 14:10:16.079908 4250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95"
dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
from {username='slave'} at 127.0.0.1:33136
I20260929 14:10:16.080411 4250 raft_consensus.cc:482] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:16.080760 4250 raft_consensus.cc:3037] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:16.114648 4250 raft_consensus.cc:504] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:16.116606 4250 leader_election.cc:288] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 4 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:16.117789 4324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:16.117738 4174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:16.118606 4324 raft_consensus.cc:3032] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:10:16.118633 4174 raft_consensus.cc:3037] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:16.119005 4324 raft_consensus.cc:729] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Running, Role: LEADER
I20260929 14:10:16.119642 4324 consensus_queue.cc:260] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:16.120537 4324 raft_consensus.cc:3037] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:16.192533 4492 raft_consensus.cc:979] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5: : Instructing follower ff2ac7ef06c44f568ec2ff0fbc58b5db to start an election
I20260929 14:10:16.192967 4498 raft_consensus.cc:1067] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 2 LEADER]: Signalling peer ff2ac7ef06c44f568ec2ff0fbc58b5db to start an election
I20260929 14:10:16.194363 4250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13"
dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
from {username='slave'} at 127.0.0.1:33136
I20260929 14:10:16.194782 4250 raft_consensus.cc:482] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:16.195145 4250 raft_consensus.cc:3037] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:16.215200 4324 raft_consensus.cc:2454] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 4.
I20260929 14:10:16.215173 4174 raft_consensus.cc:2454] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 4.
I20260929 14:10:16.215225 4250 raft_consensus.cc:504] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:16.217126 4214 leader_election.cc:302] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:16.217621 4250 leader_election.cc:288] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 3 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:16.218719 4475 raft_consensus.cc:2781] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:10:16.219049 4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:16.219620 4325 raft_consensus.cc:3032] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:10:16.219667 4174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:16.219964 4325 raft_consensus.cc:729] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Running, Role: LEADER
I20260929 14:10:16.220580 4475 raft_consensus.cc:686] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 LEADER]: Becoming Leader. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:16.220808 4325 consensus_queue.cc:260] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:16.222160 4325 raft_consensus.cc:3037] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:16.221575 4475 consensus_queue.cc:237] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:16.222402 4174 raft_consensus.cc:3037] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:16.235066 4071 catalog_manager.cc:5495] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db reported cstate change: term changed from 3 to 4, leader changed from b3e5999205bc45d3a093e26888f003f5 (127.3.242.67) to ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66). New cstate: current_term: 4 leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:16.238998 4497 raft_consensus.cc:979] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7: : Instructing follower ff2ac7ef06c44f568ec2ff0fbc58b5db to start an election
I20260929 14:10:16.239542 4494 raft_consensus.cc:1067] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Signalling peer ff2ac7ef06c44f568ec2ff0fbc58b5db to start an election
I20260929 14:10:16.241006 4250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f"
dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
from {username='slave'} at 127.0.0.1:33150
I20260929 14:10:16.241544 4250 raft_consensus.cc:482] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:16.241845 4250 raft_consensus.cc:3037] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:16.312718 4174 raft_consensus.cc:2454] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 3.
I20260929 14:10:16.312661 4325 raft_consensus.cc:2454] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 3.
I20260929 14:10:16.312685 4250 raft_consensus.cc:504] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:16.314821 4214 leader_election.cc:302] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:16.316519 4499 raft_consensus.cc:2781] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:10:16.317014 4499 raft_consensus.cc:686] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Becoming Leader. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:16.318253 4174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:16.317934 4499 consensus_queue.cc:237] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:16.318362 4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:16.318850 4174 raft_consensus.cc:3032] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:10:16.318898 4250 leader_election.cc:288] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 3 election: Requested vote from peers b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637), 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527)
I20260929 14:10:16.319155 4325 raft_consensus.cc:3037] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:16.319264 4174 raft_consensus.cc:729] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Running, Role: LEADER
I20260929 14:10:16.320499 4174 consensus_queue.cc:260] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:16.322043 4174 raft_consensus.cc:3037] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:16.326411 4071 catalog_manager.cc:5495] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db reported cstate change: term changed from 2 to 3, leader changed from b3e5999205bc45d3a093e26888f003f5 (127.3.242.67) to ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66). New cstate: current_term: 3 leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:16.374140 4497 raft_consensus.cc:979] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7: : Instructing follower ff2ac7ef06c44f568ec2ff0fbc58b5db to start an election
I20260929 14:10:16.374598 4494 raft_consensus.cc:1067] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Signalling peer ff2ac7ef06c44f568ec2ff0fbc58b5db to start an election
I20260929 14:10:16.375906 4250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d9b3984be1cf45f696793fe68b648d3f"
dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
from {username='slave'} at 127.0.0.1:33150
I20260929 14:10:16.376421 4250 raft_consensus.cc:482] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:16.376744 4250 raft_consensus.cc:3037] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:16.417794 4174 raft_consensus.cc:2454] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 3.
I20260929 14:10:16.417794 4325 raft_consensus.cc:2454] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 3.
I20260929 14:10:16.417871 4250 raft_consensus.cc:504] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:16.419534 4214 leader_election.cc:302] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:16.420114 4250 leader_election.cc:288] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 3 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:16.421136 4499 raft_consensus.cc:2781] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:10:16.421394 4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b3984be1cf45f696793fe68b648d3f" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:16.421737 4499 raft_consensus.cc:686] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Becoming Leader. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:16.421530 4174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9b3984be1cf45f696793fe68b648d3f" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:16.422147 4325 raft_consensus.cc:3037] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:16.422448 4174 raft_consensus.cc:3032] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:10:16.422853 4174 raft_consensus.cc:729] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Running, Role: LEADER
I20260929 14:10:16.422806 4499 consensus_queue.cc:237] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:16.423774 4174 consensus_queue.cc:260] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:16.425132 4174 raft_consensus.cc:3037] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:16.426798 4497 raft_consensus.cc:979] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7: : Instructing follower ff2ac7ef06c44f568ec2ff0fbc58b5db to start an election
I20260929 14:10:16.427304 4497 raft_consensus.cc:1067] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Signalling peer ff2ac7ef06c44f568ec2ff0fbc58b5db to start an election
I20260929 14:10:16.430121 4250 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34d9a8869342456ebb536c8849642565"
dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
from {username='slave'} at 127.0.0.1:33150
I20260929 14:10:16.430730 4250 raft_consensus.cc:482] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:16.431105 4250 raft_consensus.cc:3037] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:16.432101 4071 catalog_manager.cc:5495] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db reported cstate change: term changed from 2 to 3, leader changed from 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65) to ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66). New cstate: current_term: 3 leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:16.516173 4325 raft_consensus.cc:2454] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 3.
I20260929 14:10:16.516214 4250 raft_consensus.cc:504] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:16.516173 4174 raft_consensus.cc:2454] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 3.
I20260929 14:10:16.517841 4215 leader_election.cc:302] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:16.518712 4250 leader_election.cc:288] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 3 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:16.519871 4499 raft_consensus.cc:2781] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:10:16.519834 4324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d9a8869342456ebb536c8849642565" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:16.520537 4324 raft_consensus.cc:3037] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:16.520521 4499 raft_consensus.cc:686] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Becoming Leader. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:16.521247 4174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d9a8869342456ebb536c8849642565" candidate_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:16.521833 4174 raft_consensus.cc:3032] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:10:16.522239 4174 raft_consensus.cc:729] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Running, Role: LEADER
I20260929 14:10:16.521785 4499 consensus_queue.cc:237] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:16.522990 4174 consensus_queue.cc:260] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:16.524183 4174 raft_consensus.cc:3037] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:16.529078 4071 catalog_manager.cc:5495] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db reported cstate change: term changed from 2 to 3, leader changed from 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65) to ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66). New cstate: current_term: 3 leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:16.621620 4174 raft_consensus.cc:2454] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 3.
I20260929 14:10:16.621631 4324 raft_consensus.cc:2454] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff2ac7ef06c44f568ec2ff0fbc58b5db in term 3.
I20260929 14:10:16.622973 4214 leader_election.cc:302] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:16.623801 4499 raft_consensus.cc:2781] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:10:16.624307 4499 raft_consensus.cc:686] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Becoming Leader. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:16.625355 4499 consensus_queue.cc:237] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:16.633358 4071 catalog_manager.cc:5495] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db reported cstate change: term changed from 2 to 3, leader changed from 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65) to ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66). New cstate: current_term: 3 leader_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:16.669883 4324 raft_consensus.cc:1261] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 4 FOLLOWER]: Refusing update from remote peer ff2ac7ef06c44f568ec2ff0fbc58b5db: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260929 14:10:16.671051 4499 consensus_queue.cc:1035] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:10:16.680375 4174 raft_consensus.cc:1261] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 FOLLOWER]: Refusing update from remote peer ff2ac7ef06c44f568ec2ff0fbc58b5db: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260929 14:10:16.682086 4499 consensus_queue.cc:1035] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:10:16.812954 4174 raft_consensus.cc:1261] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Refusing update from remote peer ff2ac7ef06c44f568ec2ff0fbc58b5db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:16.813707 4176 raft_consensus.cc:1261] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Refusing update from remote peer ff2ac7ef06c44f568ec2ff0fbc58b5db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:16.814301 4499 consensus_queue.cc:1035] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:16.814953 4500 consensus_queue.cc:1035] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:16.825904 4324 raft_consensus.cc:1261] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Refusing update from remote peer ff2ac7ef06c44f568ec2ff0fbc58b5db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:16.828403 4325 raft_consensus.cc:1261] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Refusing update from remote peer ff2ac7ef06c44f568ec2ff0fbc58b5db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:16.830384 4499 consensus_queue.cc:1035] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:16.831352 4475 consensus_queue.cc:1035] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:16.961625 4324 raft_consensus.cc:1261] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Refusing update from remote peer ff2ac7ef06c44f568ec2ff0fbc58b5db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:16.962867 4475 consensus_queue.cc:1035] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:16.971920 4174 raft_consensus.cc:1261] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Refusing update from remote peer ff2ac7ef06c44f568ec2ff0fbc58b5db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:16.974123 4499 consensus_queue.cc:1035] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:17.081636 4324 raft_consensus.cc:1261] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Refusing update from remote peer ff2ac7ef06c44f568ec2ff0fbc58b5db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:17.082801 4499 consensus_queue.cc:1035] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:17.091486 4174 raft_consensus.cc:1261] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Refusing update from remote peer ff2ac7ef06c44f568ec2ff0fbc58b5db: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:17.093300 4475 consensus_queue.cc:1035] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260929 14:10:19.000187 4041 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260929 14:10:19.000965 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:19.010593 4250 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0a3a6ae8c5f44410af8b79ac8a1de584"
dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
mode: GRACEFUL
new_leader_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
from {username='slave'} at 127.0.0.1:33162
I20260929 14:10:19.011227 4250 raft_consensus.cc:595] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 LEADER]: Received request to transfer leadership to TS 55e23cb99a754a72b1144e3e226b1eb7
I20260929 14:10:19.013967 4250 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "34d9a8869342456ebb536c8849642565"
dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
mode: GRACEFUL
new_leader_uuid: "b3e5999205bc45d3a093e26888f003f5"
from {username='slave'} at 127.0.0.1:33162
I20260929 14:10:19.014343 4250 raft_consensus.cc:595] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Received request to transfer leadership to TS b3e5999205bc45d3a093e26888f003f5
I20260929 14:10:19.016408 4250 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95"
dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
mode: GRACEFUL
new_leader_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
from {username='slave'} at 127.0.0.1:33162
I20260929 14:10:19.016775 4250 raft_consensus.cc:595] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 LEADER]: Received request to transfer leadership to TS 55e23cb99a754a72b1144e3e226b1eb7
I20260929 14:10:19.018867 4250 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13"
dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
mode: GRACEFUL
new_leader_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
from {username='slave'} at 127.0.0.1:33162
I20260929 14:10:19.019379 4250 raft_consensus.cc:595] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Received request to transfer leadership to TS 55e23cb99a754a72b1144e3e226b1eb7
I20260929 14:10:19.022151 4250 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f"
dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
mode: GRACEFUL
new_leader_uuid: "b3e5999205bc45d3a093e26888f003f5"
from {username='slave'} at 127.0.0.1:33162
I20260929 14:10:19.022662 4250 raft_consensus.cc:595] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Received request to transfer leadership to TS b3e5999205bc45d3a093e26888f003f5
I20260929 14:10:19.023598 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260929 14:10:19.024163 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:19.024942 4526 raft_consensus.cc:979] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db: : Instructing follower 55e23cb99a754a72b1144e3e226b1eb7 to start an election
I20260929 14:10:19.025391 4526 raft_consensus.cc:1067] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 LEADER]: Signalling peer 55e23cb99a754a72b1144e3e226b1eb7 to start an election
I20260929 14:10:19.027124 4174 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95"
dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
from {username='slave'} at 127.0.0.1:34814
I20260929 14:10:19.027575 4174 raft_consensus.cc:482] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:19.027900 4174 raft_consensus.cc:3037] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:10:19.052738 4174 raft_consensus.cc:504] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:19.055599 4174 leader_election.cc:288] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 5 election: Requested vote from peers ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:19.056764 4250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:19.056985 4324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57f069c685a74d38ae9570dfec76ad95" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:19.057600 4250 raft_consensus.cc:3032] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:10:19.057678 4324 raft_consensus.cc:3037] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:10:19.057894 4250 raft_consensus.cc:729] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:19.058725 4250 consensus_queue.cc:260] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:19.059749 4250 raft_consensus.cc:3037] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:10:19.079522 4500 raft_consensus.cc:979] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db: : Instructing follower 55e23cb99a754a72b1144e3e226b1eb7 to start an election
I20260929 14:10:19.079957 4500 raft_consensus.cc:1067] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Signalling peer 55e23cb99a754a72b1144e3e226b1eb7 to start an election
I20260929 14:10:19.081291 4174 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13"
dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
from {username='slave'} at 127.0.0.1:34814
I20260929 14:10:19.081730 4174 raft_consensus.cc:482] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:19.081974 4174 raft_consensus.cc:3037] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:19.109823 4526 raft_consensus.cc:979] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db: : Instructing follower b3e5999205bc45d3a093e26888f003f5 to start an election
I20260929 14:10:19.110261 4526 raft_consensus.cc:1067] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Signalling peer b3e5999205bc45d3a093e26888f003f5 to start an election
I20260929 14:10:19.111778 4325 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34d9a8869342456ebb536c8849642565"
dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
from {username='slave'} at 127.0.0.1:52930
I20260929 14:10:19.112262 4325 raft_consensus.cc:482] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:19.112540 4325 raft_consensus.cc:3037] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:19.157276 4324 raft_consensus.cc:2454] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e23cb99a754a72b1144e3e226b1eb7 in term 5.
I20260929 14:10:19.157398 4174 raft_consensus.cc:504] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:19.157276 4250 raft_consensus.cc:2454] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e23cb99a754a72b1144e3e226b1eb7 in term 5.
I20260929 14:10:19.159880 4174 leader_election.cc:288] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 4 election: Requested vote from peers ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:19.160336 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:19.160961 4249 raft_consensus.cc:3032] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:10:19.161278 4249 raft_consensus.cc:729] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:19.161250 4140 leader_election.cc:302] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, b3e5999205bc45d3a093e26888f003f5; no voters:
I20260929 14:10:19.162199 4249 consensus_queue.cc:260] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:19.163466 4249 raft_consensus.cc:3037] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:19.163335 4324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b0afbf24ceb462cbc079ccfbec60c13" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:19.163861 4324 raft_consensus.cc:3037] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:19.165326 4527 raft_consensus.cc:2781] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:10:19.166402 4527 raft_consensus.cc:686] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 5 LEADER]: Becoming Leader. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Running, Role: LEADER
I20260929 14:10:19.167296 4527 consensus_queue.cc:237] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:19.175182 4071 catalog_manager.cc:5495] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 reported cstate change: term changed from 4 to 5, leader changed from ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66) to 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65). New cstate: current_term: 5 leader_uuid: "55e23cb99a754a72b1144e3e226b1eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:19.260133 4249 raft_consensus.cc:2454] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e23cb99a754a72b1144e3e226b1eb7 in term 4.
I20260929 14:10:19.260370 4325 raft_consensus.cc:504] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:19.261704 4141 leader_election.cc:302] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:19.262769 4527 raft_consensus.cc:2781] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:10:19.263336 4527 raft_consensus.cc:686] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 LEADER]: Becoming Leader. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Running, Role: LEADER
I20260929 14:10:19.264499 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d9a8869342456ebb536c8849642565" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:19.264257 4527 consensus_queue.cc:237] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:19.265019 4249 raft_consensus.cc:3032] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:10:19.265383 4249 raft_consensus.cc:729] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:19.265322 4174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34d9a8869342456ebb536c8849642565" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:19.265930 4174 raft_consensus.cc:3037] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:19.266148 4249 consensus_queue.cc:260] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:19.267146 4249 raft_consensus.cc:3037] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:19.267347 4324 raft_consensus.cc:2454] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e23cb99a754a72b1144e3e226b1eb7 in term 4.
I20260929 14:10:19.269026 4325 leader_election.cc:288] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 4 election: Requested vote from peers 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527), ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039)
I20260929 14:10:19.274533 4249 raft_consensus.cc:2454] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 4.
I20260929 14:10:19.274643 4174 raft_consensus.cc:2454] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 4.
I20260929 14:10:19.275660 4290 leader_election.cc:302] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3e5999205bc45d3a093e26888f003f5, ff2ac7ef06c44f568ec2ff0fbc58b5db; no voters:
I20260929 14:10:19.275409 4071 catalog_manager.cc:5495] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 reported cstate change: term changed from 3 to 4, leader changed from ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66) to 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65). New cstate: current_term: 4 leader_uuid: "55e23cb99a754a72b1144e3e226b1eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:19.276497 4498 raft_consensus.cc:2781] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:10:19.277638 4498 raft_consensus.cc:686] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 4 LEADER]: Becoming Leader. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Running, Role: LEADER
I20260929 14:10:19.278398 4498 consensus_queue.cc:237] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:19.285400 4071 catalog_manager.cc:5495] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 reported cstate change: term changed from 3 to 4, leader changed from ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66) to b3e5999205bc45d3a093e26888f003f5 (127.3.242.67). New cstate: current_term: 4 leader_uuid: "b3e5999205bc45d3a093e26888f003f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: HEALTHY } } }
I20260929 14:10:19.456988 4500 raft_consensus.cc:979] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db: : Instructing follower b3e5999205bc45d3a093e26888f003f5 to start an election
I20260929 14:10:19.457512 4526 raft_consensus.cc:1067] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Signalling peer b3e5999205bc45d3a093e26888f003f5 to start an election
I20260929 14:10:19.458788 4325 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f"
dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
from {username='slave'} at 127.0.0.1:52930
I20260929 14:10:19.459254 4325 raft_consensus.cc:482] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:19.459527 4325 raft_consensus.cc:3037] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:19.465085 4325 raft_consensus.cc:504] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:19.466817 4325 leader_election.cc:288] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 4 election: Requested vote from peers ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039), 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527)
I20260929 14:10:19.467995 4174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
I20260929 14:10:19.467795 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f71ae1c4a24a1cb89f70436ac6d39f" candidate_uuid: "b3e5999205bc45d3a093e26888f003f5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:19.468405 4174 raft_consensus.cc:3037] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:19.468467 4249 raft_consensus.cc:3032] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:10:19.468812 4249 raft_consensus.cc:729] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:19.469411 4249 consensus_queue.cc:260] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:19.470456 4249 raft_consensus.cc:3037] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:19.473002 4174 raft_consensus.cc:2454] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 4.
I20260929 14:10:19.474253 4288 leader_election.cc:302] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, b3e5999205bc45d3a093e26888f003f5; no voters:
I20260929 14:10:19.475148 4498 raft_consensus.cc:2781] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:10:19.475286 4249 raft_consensus.cc:2454] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3e5999205bc45d3a093e26888f003f5 in term 4.
I20260929 14:10:19.475636 4498 raft_consensus.cc:686] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 4 LEADER]: Becoming Leader. State: Replica: b3e5999205bc45d3a093e26888f003f5, State: Running, Role: LEADER
I20260929 14:10:19.476504 4498 consensus_queue.cc:237] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } }
I20260929 14:10:19.483116 4071 catalog_manager.cc:5495] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 reported cstate change: term changed from 3 to 4, leader changed from ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66) to b3e5999205bc45d3a093e26888f003f5 (127.3.242.67). New cstate: current_term: 4 leader_uuid: "b3e5999205bc45d3a093e26888f003f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:19.499599 4500 raft_consensus.cc:979] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db: : Instructing follower 55e23cb99a754a72b1144e3e226b1eb7 to start an election
I20260929 14:10:19.500044 4526 raft_consensus.cc:1067] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 LEADER]: Signalling peer 55e23cb99a754a72b1144e3e226b1eb7 to start an election
I20260929 14:10:19.501384 4174 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0a3a6ae8c5f44410af8b79ac8a1de584"
dest_uuid: "55e23cb99a754a72b1144e3e226b1eb7"
from {username='slave'} at 127.0.0.1:34814
I20260929 14:10:19.501801 4174 raft_consensus.cc:482] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:19.502056 4174 raft_consensus.cc:3037] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:19.505817 4174 raft_consensus.cc:504] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:19.507053 4174 leader_election.cc:288] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 3 election: Requested vote from peers ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66:43039), b3e5999205bc45d3a093e26888f003f5 (127.3.242.67:45637)
I20260929 14:10:19.507977 4325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3a6ae8c5f44410af8b79ac8a1de584" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b3e5999205bc45d3a093e26888f003f5"
I20260929 14:10:19.507932 4249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a3a6ae8c5f44410af8b79ac8a1de584" candidate_uuid: "55e23cb99a754a72b1144e3e226b1eb7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db"
I20260929 14:10:19.508603 4325 raft_consensus.cc:3037] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:19.508651 4249 raft_consensus.cc:3032] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:10:19.509099 4249 raft_consensus.cc:729] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ff2ac7ef06c44f568ec2ff0fbc58b5db, State: Running, Role: LEADER
I20260929 14:10:19.509806 4249 consensus_queue.cc:260] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:19.510999 4249 raft_consensus.cc:3037] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:19.514955 4325 raft_consensus.cc:2454] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e23cb99a754a72b1144e3e226b1eb7 in term 3.
I20260929 14:10:19.516162 4140 leader_election.cc:302] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55e23cb99a754a72b1144e3e226b1eb7, b3e5999205bc45d3a093e26888f003f5; no voters:
I20260929 14:10:19.516649 4249 raft_consensus.cc:2454] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55e23cb99a754a72b1144e3e226b1eb7 in term 3.
I20260929 14:10:19.516831 4527 raft_consensus.cc:2781] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:10:19.517354 4527 raft_consensus.cc:686] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 LEADER]: Becoming Leader. State: Replica: 55e23cb99a754a72b1144e3e226b1eb7, State: Running, Role: LEADER
I20260929 14:10:19.518178 4527 consensus_queue.cc:237] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } }
I20260929 14:10:19.524993 4071 catalog_manager.cc:5495] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 reported cstate change: term changed from 2 to 3, leader changed from ff2ac7ef06c44f568ec2ff0fbc58b5db (127.3.242.66) to 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65). New cstate: current_term: 3 leader_uuid: "55e23cb99a754a72b1144e3e226b1eb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:19.713244 4325 raft_consensus.cc:1261] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 5 FOLLOWER]: Refusing update from remote peer 55e23cb99a754a72b1144e3e226b1eb7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260929 14:10:19.714524 4527 consensus_queue.cc:1035] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:10:19.714944 4249 raft_consensus.cc:1261] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 5 FOLLOWER]: Refusing update from remote peer 55e23cb99a754a72b1144e3e226b1eb7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260929 14:10:19.716075 4527 consensus_queue.cc:1035] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:10:19.764950 4249 raft_consensus.cc:1261] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 FOLLOWER]: Refusing update from remote peer b3e5999205bc45d3a093e26888f003f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260929 14:10:19.766042 4498 consensus_queue.cc:1035] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:10:19.774528 4174 raft_consensus.cc:1261] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 FOLLOWER]: Refusing update from remote peer b3e5999205bc45d3a093e26888f003f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260929 14:10:19.776239 4498 consensus_queue.cc:1035] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:10:19.791306 4325 raft_consensus.cc:1261] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 4 FOLLOWER]: Refusing update from remote peer 55e23cb99a754a72b1144e3e226b1eb7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260929 14:10:19.792702 4528 consensus_queue.cc:1035] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:10:19.802723 4249 raft_consensus.cc:1261] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 FOLLOWER]: Refusing update from remote peer 55e23cb99a754a72b1144e3e226b1eb7: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260929 14:10:19.804095 4528 consensus_queue.cc:1035] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:10:19.883817 4249 raft_consensus.cc:1261] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 FOLLOWER]: Refusing update from remote peer b3e5999205bc45d3a093e26888f003f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260929 14:10:19.885036 4498 consensus_queue.cc:1035] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:10:19.895931 4174 raft_consensus.cc:1261] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 FOLLOWER]: Refusing update from remote peer b3e5999205bc45d3a093e26888f003f5: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260929 14:10:19.897547 4498 consensus_queue.cc:1035] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55e23cb99a754a72b1144e3e226b1eb7" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 42527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:10:20.021879 4325 raft_consensus.cc:1261] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Refusing update from remote peer 55e23cb99a754a72b1144e3e226b1eb7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:20.023085 4528 consensus_queue.cc:1035] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3e5999205bc45d3a093e26888f003f5" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 45637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:20.025136 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:20.033334 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:20.033941 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:20.035063 4249 raft_consensus.cc:1261] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Refusing update from remote peer 55e23cb99a754a72b1144e3e226b1eb7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:20.036587 4528 consensus_queue.cc:1035] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff2ac7ef06c44f568ec2ff0fbc58b5db" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:21.034842 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:21.040524 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:21.040987 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:22.042127 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:22.046806 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:22.047235 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:23.048074 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:23.053505 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:23.053998 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:24.054840 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:24.059748 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:24.060181 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:25.061007 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:25.065732 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:25.066150 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:26.066875 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:26.072151 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:26.072597 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:27.073494 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:27.080233 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:27.080832 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:28.081686 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:28.086468 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:28.086915 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:29.120409 4041 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260929 14:10:29.121428 4041 tablet_server.cc:178] TabletServer@127.3.242.65:0 shutting down...
W20260929 14:10:29.135717 4214 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260929 14:10:29.141058 4214 consensus_peers.cc:487] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db -> Peer 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527): Couldn't send request to peer 55e23cb99a754a72b1144e3e226b1eb7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:10:29.157464 4041 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260929 14:10:29.158226 4041 tablet_replica.cc:330] T 26d46ca3672f4153b12484ab0eecbfe7 P 55e23cb99a754a72b1144e3e226b1eb7: stopping tablet replica
I20260929 14:10:29.158969 4041 raft_consensus.cc:2229] T 26d46ca3672f4153b12484ab0eecbfe7 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.159682 4041 raft_consensus.cc:2258] T 26d46ca3672f4153b12484ab0eecbfe7 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.161971 4041 tablet_replica.cc:330] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7: stopping tablet replica
I20260929 14:10:29.162526 4041 raft_consensus.cc:2229] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.163002 4041 raft_consensus.cc:2258] T d9b3984be1cf45f696793fe68b648d3f P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.164638 4041 tablet_replica.cc:330] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7: stopping tablet replica
I20260929 14:10:29.165127 4041 raft_consensus.cc:2229] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 LEADER]: Raft consensus shutting down.
I20260929 14:10:29.165966 4041 raft_consensus.cc:2258] T 5b0afbf24ceb462cbc079ccfbec60c13 P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.167537 4041 tablet_replica.cc:330] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7: stopping tablet replica
I20260929 14:10:29.168009 4041 raft_consensus.cc:2229] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.168500 4041 raft_consensus.cc:2258] T f4f71ae1c4a24a1cb89f70436ac6d39f P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.170045 4041 tablet_replica.cc:330] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7: stopping tablet replica
I20260929 14:10:29.170668 4041 raft_consensus.cc:2229] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.171044 4041 raft_consensus.cc:2258] T e52a714cc5d542fdadb1fb29c39392d1 P 55e23cb99a754a72b1144e3e226b1eb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.172600 4041 tablet_replica.cc:330] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7: stopping tablet replica
I20260929 14:10:29.173053 4041 raft_consensus.cc:2229] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 LEADER]: Raft consensus shutting down.
I20260929 14:10:29.173776 4041 raft_consensus.cc:2258] T 0a3a6ae8c5f44410af8b79ac8a1de584 P 55e23cb99a754a72b1144e3e226b1eb7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.175271 4041 tablet_replica.cc:330] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7: stopping tablet replica
I20260929 14:10:29.175711 4041 raft_consensus.cc:2229] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.176146 4041 raft_consensus.cc:2258] T 34d9a8869342456ebb536c8849642565 P 55e23cb99a754a72b1144e3e226b1eb7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.177826 4041 tablet_replica.cc:330] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7: stopping tablet replica
I20260929 14:10:29.178256 4041 raft_consensus.cc:2229] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 5 LEADER]: Raft consensus shutting down.
I20260929 14:10:29.179145 4041 raft_consensus.cc:2258] T 57f069c685a74d38ae9570dfec76ad95 P 55e23cb99a754a72b1144e3e226b1eb7 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.181022 4041 tablet_replica.cc:330] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7: stopping tablet replica
I20260929 14:10:29.181504 4041 raft_consensus.cc:2229] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.181982 4041 raft_consensus.cc:2258] T 118c8454885a48248a988813bdba96cc P 55e23cb99a754a72b1144e3e226b1eb7 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260929 14:10:29.183483 4288 consensus_peers.cc:487] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 -> Peer 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527): Couldn't send request to peer 55e23cb99a754a72b1144e3e226b1eb7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260929 14:10:29.296525 4214 consensus_peers.cc:487] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db -> Peer 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527): Couldn't send request to peer 55e23cb99a754a72b1144e3e226b1eb7. Status: Network error: Client connection negotiation failed: client connection to 127.3.242.65:42527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 14:10:29.297032 4041 tablet_server.cc:194] TabletServer@127.3.242.65:0 shutdown complete.
W20260929 14:10:29.307888 4288 consensus_peers.cc:487] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 -> Peer 55e23cb99a754a72b1144e3e226b1eb7 (127.3.242.65:42527): Couldn't send request to peer 55e23cb99a754a72b1144e3e226b1eb7. Status: Network error: Client connection negotiation failed: client connection to 127.3.242.65:42527: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 14:10:29.319710 4041 tablet_server.cc:178] TabletServer@127.3.242.66:0 shutting down...
I20260929 14:10:29.344903 4041 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260929 14:10:29.345634 4041 tablet_replica.cc:330] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db: stopping tablet replica
I20260929 14:10:29.346277 4041 raft_consensus.cc:2229] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.346822 4041 raft_consensus.cc:2258] T 118c8454885a48248a988813bdba96cc P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.348532 4041 tablet_replica.cc:330] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db: stopping tablet replica
I20260929 14:10:29.349004 4041 raft_consensus.cc:2229] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:10:29.349745 4041 raft_consensus.cc:2258] T e52a714cc5d542fdadb1fb29c39392d1 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.351565 4041 tablet_replica.cc:330] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db: stopping tablet replica
I20260929 14:10:29.352227 4041 raft_consensus.cc:2229] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:10:29.353083 4041 raft_consensus.cc:2258] T 26d46ca3672f4153b12484ab0eecbfe7 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.355137 4041 tablet_replica.cc:330] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db: stopping tablet replica
I20260929 14:10:29.355902 4041 raft_consensus.cc:2229] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 LEADER]: Raft consensus shutting down.
I20260929 14:10:29.356884 4041 raft_consensus.cc:2258] T d9b3984be1cf45f696793fe68b648d3f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.358546 4041 tablet_replica.cc:330] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db: stopping tablet replica
I20260929 14:10:29.359007 4041 raft_consensus.cc:2229] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.359390 4041 raft_consensus.cc:2258] T 34d9a8869342456ebb536c8849642565 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.361011 4041 tablet_replica.cc:330] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db: stopping tablet replica
I20260929 14:10:29.361512 4041 raft_consensus.cc:2229] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.361912 4041 raft_consensus.cc:2258] T 0a3a6ae8c5f44410af8b79ac8a1de584 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 3 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.363933 4041 tablet_replica.cc:330] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db: stopping tablet replica
I20260929 14:10:29.364511 4041 raft_consensus.cc:2229] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 5 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.364993 4041 raft_consensus.cc:2258] T 57f069c685a74d38ae9570dfec76ad95 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 5 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.367033 4041 tablet_replica.cc:330] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db: stopping tablet replica
I20260929 14:10:29.367539 4041 raft_consensus.cc:2229] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.367957 4041 raft_consensus.cc:2258] T 5b0afbf24ceb462cbc079ccfbec60c13 P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.369446 4041 tablet_replica.cc:330] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db: stopping tablet replica
I20260929 14:10:29.369936 4041 raft_consensus.cc:2229] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.370394 4041 raft_consensus.cc:2258] T f4f71ae1c4a24a1cb89f70436ac6d39f P ff2ac7ef06c44f568ec2ff0fbc58b5db [term 4 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.397504 4041 tablet_server.cc:194] TabletServer@127.3.242.66:0 shutdown complete.
I20260929 14:10:29.416289 4041 tablet_server.cc:178] TabletServer@127.3.242.67:0 shutting down...
I20260929 14:10:29.440661 4041 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260929 14:10:29.441426 4041 tablet_replica.cc:330] T e52a714cc5d542fdadb1fb29c39392d1 P b3e5999205bc45d3a093e26888f003f5: stopping tablet replica
I20260929 14:10:29.442101 4041 raft_consensus.cc:2229] T e52a714cc5d542fdadb1fb29c39392d1 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.442553 4041 raft_consensus.cc:2258] T e52a714cc5d542fdadb1fb29c39392d1 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.444404 4041 tablet_replica.cc:330] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5: stopping tablet replica
I20260929 14:10:29.445019 4041 raft_consensus.cc:2229] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [term 2 LEADER]: Raft consensus shutting down.
I20260929 14:10:29.445896 4041 raft_consensus.cc:2258] T 118c8454885a48248a988813bdba96cc P b3e5999205bc45d3a093e26888f003f5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.448233 4041 tablet_replica.cc:330] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5: stopping tablet replica
I20260929 14:10:29.448794 4041 raft_consensus.cc:2229] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 4 LEADER]: Raft consensus shutting down.
I20260929 14:10:29.449637 4041 raft_consensus.cc:2258] T 34d9a8869342456ebb536c8849642565 P b3e5999205bc45d3a093e26888f003f5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.451874 4041 tablet_replica.cc:330] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5: stopping tablet replica
I20260929 14:10:29.452498 4041 raft_consensus.cc:2229] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.452965 4041 raft_consensus.cc:2258] T 0a3a6ae8c5f44410af8b79ac8a1de584 P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.454679 4041 tablet_replica.cc:330] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5: stopping tablet replica
I20260929 14:10:29.455188 4041 raft_consensus.cc:2229] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 4 LEADER]: Raft consensus shutting down.
I20260929 14:10:29.456074 4041 raft_consensus.cc:2258] T f4f71ae1c4a24a1cb89f70436ac6d39f P b3e5999205bc45d3a093e26888f003f5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.457865 4041 tablet_replica.cc:330] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5: stopping tablet replica
I20260929 14:10:29.458379 4041 raft_consensus.cc:2229] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.458767 4041 raft_consensus.cc:2258] T 5b0afbf24ceb462cbc079ccfbec60c13 P b3e5999205bc45d3a093e26888f003f5 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.460319 4041 tablet_replica.cc:330] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5: stopping tablet replica
I20260929 14:10:29.460788 4041 raft_consensus.cc:2229] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.461172 4041 raft_consensus.cc:2258] T 57f069c685a74d38ae9570dfec76ad95 P b3e5999205bc45d3a093e26888f003f5 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.463878 4041 tablet_replica.cc:330] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5: stopping tablet replica
I20260929 14:10:29.464318 4041 raft_consensus.cc:2229] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.464671 4041 raft_consensus.cc:2258] T d9b3984be1cf45f696793fe68b648d3f P b3e5999205bc45d3a093e26888f003f5 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.466379 4041 tablet_replica.cc:330] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5: stopping tablet replica
I20260929 14:10:29.466755 4041 raft_consensus.cc:2229] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:29.467099 4041 raft_consensus.cc:2258] T 26d46ca3672f4153b12484ab0eecbfe7 P b3e5999205bc45d3a093e26888f003f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.494277 4041 tablet_server.cc:194] TabletServer@127.3.242.67:0 shutdown complete.
I20260929 14:10:29.512844 4041 master.cc:537] Master@127.3.242.126:37097 shutting down...
I20260929 14:10:29.639283 4041 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:10:29.639948 4041 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:29.640317 4041 tablet_replica.cc:330] T 00000000000000000000000000000000 P 42be7bc87f17483a96850995c458fb2f: stopping tablet replica
I20260929 14:10:29.659680 4041 master.cc:552] Master@127.3.242.126:37097 shutdown complete.
55e23cb99a754a72b1144e3e226b1eb7 4
b3e5999205bc45d3a093e26888f003f5 1
ff2ac7ef06c44f568ec2ff0fbc58b5db 4
55e23cb99a754a72b1144e3e226b1eb7 3
b3e5999205bc45d3a093e26888f003f5 3
ff2ac7ef06c44f568ec2ff0fbc58b5db 3
55e23cb99a754a72b1144e3e226b1eb7 0
b3e5999205bc45d3a093e26888f003f5 1
ff2ac7ef06c44f568ec2ff0fbc58b5db 8
55e23cb99a754a72b1144e3e226b1eb7 3
b3e5999205bc45d3a093e26888f003f5 3
ff2ac7ef06c44f568ec2ff0fbc58b5db 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (31020 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260929 14:10:29.701596 4041 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.242.126:40359
I20260929 14:10:29.702694 4041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:10:29.708714 4568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:29.709367 4569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:29.709898 4571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:10:29.710747 4041 server_base.cc:1034] running on GCE node
I20260929 14:10:29.711694 4041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 14:10:29.711907 4041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 14:10:29.712105 4041 hybrid_clock.cc:648] HybridClock initialized: now 1790691029712072 us; error 0 us; skew 500 ppm
I20260929 14:10:29.712651 4041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 14:10:29.719393 4041 webserver.cc:460] Webserver started at http://127.3.242.126:34837/ using document root <none> and password file <none>
I20260929 14:10:29.719921 4041 fs_manager.cc:359] Metadata directory not provided
I20260929 14:10:29.720104 4041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:10:29.720422 4041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:10:29.721913 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/master-0-root/instance:
uuid: "225b6703403a47c8a9c887c7f451b0a2"
format_stamp: "Formatted at 2026-09-29 14:10:29 on dist-test-slave-6gv9"
I20260929 14:10:29.726606 4041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:10:29.730067 4576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:29.730880 4041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:10:29.731181 4041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/master-0-root
uuid: "225b6703403a47c8a9c887c7f451b0a2"
format_stamp: "Formatted at 2026-09-29 14:10:29 on dist-test-slave-6gv9"
I20260929 14:10:29.731463 4041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:10:29.750113 4041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:10:29.751430 4041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:10:29.791177 4041 rpc_server.cc:307] RPC server started. Bound to: 127.3.242.126:40359
I20260929 14:10:29.791280 4627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.242.126:40359 every 8 connection(s)
I20260929 14:10:29.795162 4628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:29.804805 4628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2: Bootstrap starting.
I20260929 14:10:29.809273 4628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:29.813557 4628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2: No bootstrap required, opened a new log
I20260929 14:10:29.815546 4628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225b6703403a47c8a9c887c7f451b0a2" member_type: VOTER }
I20260929 14:10:29.815984 4628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:29.816298 4628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 225b6703403a47c8a9c887c7f451b0a2, State: Initialized, Role: FOLLOWER
I20260929 14:10:29.816947 4628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225b6703403a47c8a9c887c7f451b0a2" member_type: VOTER }
I20260929 14:10:29.817472 4628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:10:29.817698 4628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:10:29.818002 4628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:29.821698 4628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225b6703403a47c8a9c887c7f451b0a2" member_type: VOTER }
I20260929 14:10:29.822196 4628 leader_election.cc:302] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 225b6703403a47c8a9c887c7f451b0a2; no voters:
I20260929 14:10:29.823273 4628 leader_election.cc:288] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:10:29.823611 4631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:29.824930 4631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [term 1 LEADER]: Becoming Leader. State: Replica: 225b6703403a47c8a9c887c7f451b0a2, State: Running, Role: LEADER
I20260929 14:10:29.825626 4631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225b6703403a47c8a9c887c7f451b0a2" member_type: VOTER }
I20260929 14:10:29.826251 4628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:10:29.830662 4633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 225b6703403a47c8a9c887c7f451b0a2. Latest consensus state: current_term: 1 leader_uuid: "225b6703403a47c8a9c887c7f451b0a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225b6703403a47c8a9c887c7f451b0a2" member_type: VOTER } }
I20260929 14:10:29.830856 4632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "225b6703403a47c8a9c887c7f451b0a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "225b6703403a47c8a9c887c7f451b0a2" member_type: VOTER } }
I20260929 14:10:29.831615 4633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [sys.catalog]: This master's current role is: LEADER
I20260929 14:10:29.831606 4632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [sys.catalog]: This master's current role is: LEADER
I20260929 14:10:29.835320 4639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:10:29.840256 4639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:10:29.846346 4041 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260929 14:10:29.849810 4639 catalog_manager.cc:1347] Generated new cluster ID: 712562839e1d4914ada86232d7d74b3c
I20260929 14:10:29.850092 4639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:10:29.874236 4639 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:10:29.875912 4639 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:10:29.892459 4639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2: Generated new TSK 0
I20260929 14:10:29.893111 4639 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:10:29.913066 4041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:10:29.919078 4649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:29.920254 4650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:29.922478 4652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:10:29.923398 4041 server_base.cc:1034] running on GCE node
I20260929 14:10:29.924294 4041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 14:10:29.924461 4041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 14:10:29.924572 4041 hybrid_clock.cc:648] HybridClock initialized: now 1790691029924564 us; error 0 us; skew 500 ppm
I20260929 14:10:29.925004 4041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 14:10:29.927871 4041 webserver.cc:460] Webserver started at http://127.3.242.65:38031/ using document root <none> and password file <none>
I20260929 14:10:29.928334 4041 fs_manager.cc:359] Metadata directory not provided
I20260929 14:10:29.928488 4041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:10:29.928697 4041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:10:29.929821 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/ts-0-root/instance:
uuid: "5898905d9cda47a88faa1bc424bad7ae"
format_stamp: "Formatted at 2026-09-29 14:10:29 on dist-test-slave-6gv9"
I20260929 14:10:29.934423 4041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:10:29.937469 4657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:29.938233 4041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:10:29.938512 4041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/ts-0-root
uuid: "5898905d9cda47a88faa1bc424bad7ae"
format_stamp: "Formatted at 2026-09-29 14:10:29 on dist-test-slave-6gv9"
I20260929 14:10:29.938817 4041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:10:29.950443 4041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:10:29.951604 4041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:10:29.953042 4041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:10:29.955227 4041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:10:29.955448 4041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:29.955687 4041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:10:29.955924 4041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:29.994841 4041 rpc_server.cc:307] RPC server started. Bound to: 127.3.242.65:35523
I20260929 14:10:29.994966 4719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.242.65:35523 every 8 connection(s)
I20260929 14:10:29.999261 4041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:10:30.007493 4724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:30.007665 4725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:10:30.012998 4041 server_base.cc:1034] running on GCE node
W20260929 14:10:30.013196 4727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:10:30.014384 4041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 14:10:30.014627 4041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 14:10:30.014864 4041 hybrid_clock.cc:648] HybridClock initialized: now 1790691030014845 us; error 0 us; skew 500 ppm
I20260929 14:10:30.015496 4041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 14:10:30.017180 4720 heartbeater.cc:347] Connected to a master server at 127.3.242.126:40359
I20260929 14:10:30.017653 4720 heartbeater.cc:464] Registering TS with master...
I20260929 14:10:30.018473 4720 heartbeater.cc:511] Master 127.3.242.126:40359 requested a full tablet report, sending...
I20260929 14:10:30.018633 4041 webserver.cc:460] Webserver started at http://127.3.242.66:44601/ using document root <none> and password file <none>
I20260929 14:10:30.019210 4041 fs_manager.cc:359] Metadata directory not provided
I20260929 14:10:30.019440 4041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:10:30.019761 4041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:10:30.020686 4593 ts_manager.cc:194] Registered new tserver with Master: 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523)
I20260929 14:10:30.020875 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/ts-1-root/instance:
uuid: "376e574c0386490bbc10b1c68119b938"
format_stamp: "Formatted at 2026-09-29 14:10:30 on dist-test-slave-6gv9"
I20260929 14:10:30.022534 4593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44352
I20260929 14:10:30.026010 4041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:10:30.029438 4732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:30.030429 4041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:10:30.030756 4041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/ts-1-root
uuid: "376e574c0386490bbc10b1c68119b938"
format_stamp: "Formatted at 2026-09-29 14:10:30 on dist-test-slave-6gv9"
I20260929 14:10:30.031075 4041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:10:30.050504 4041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:10:30.051700 4041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:10:30.053464 4041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:10:30.055670 4041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:10:30.055896 4041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:30.056181 4041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:10:30.056353 4041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:30.097155 4041 rpc_server.cc:307] RPC server started. Bound to: 127.3.242.66:42601
I20260929 14:10:30.097340 4794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.242.66:42601 every 8 connection(s)
I20260929 14:10:30.102134 4041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:10:30.109118 4798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:30.111760 4799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:30.112716 4801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:10:30.112742 4041 server_base.cc:1034] running on GCE node
I20260929 14:10:30.113900 4041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 14:10:30.114086 4041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 14:10:30.114295 4041 hybrid_clock.cc:648] HybridClock initialized: now 1790691030114261 us; error 0 us; skew 500 ppm
I20260929 14:10:30.114996 4041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 14:10:30.117959 4041 webserver.cc:460] Webserver started at http://127.3.242.67:36273/ using document root <none> and password file <none>
I20260929 14:10:30.118507 4041 fs_manager.cc:359] Metadata directory not provided
I20260929 14:10:30.118757 4041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:10:30.119071 4041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:10:30.120414 4795 heartbeater.cc:347] Connected to a master server at 127.3.242.126:40359
I20260929 14:10:30.120573 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/ts-2-root/instance:
uuid: "dab47308abbe40bcb7bb9eb70781192a"
format_stamp: "Formatted at 2026-09-29 14:10:30 on dist-test-slave-6gv9"
I20260929 14:10:30.120769 4795 heartbeater.cc:464] Registering TS with master...
I20260929 14:10:30.121533 4795 heartbeater.cc:511] Master 127.3.242.126:40359 requested a full tablet report, sending...
I20260929 14:10:30.123425 4593 ts_manager.cc:194] Registered new tserver with Master: 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601)
I20260929 14:10:30.125495 4593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44354
I20260929 14:10:30.126169 4041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 14:10:30.130232 4806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:30.130962 4041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260929 14:10:30.131250 4041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/ts-2-root
uuid: "dab47308abbe40bcb7bb9eb70781192a"
format_stamp: "Formatted at 2026-09-29 14:10:30 on dist-test-slave-6gv9"
I20260929 14:10:30.131542 4041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1790690998520575-4041-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:10:30.157059 4041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:10:30.158212 4041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:10:30.159615 4041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:10:30.161671 4041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:10:30.161907 4041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:30.162144 4041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:10:30.162312 4041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:30.199543 4041 rpc_server.cc:307] RPC server started. Bound to: 127.3.242.67:40873
I20260929 14:10:30.199643 4868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.242.67:40873 every 8 connection(s)
I20260929 14:10:30.216825 4869 heartbeater.cc:347] Connected to a master server at 127.3.242.126:40359
I20260929 14:10:30.217200 4869 heartbeater.cc:464] Registering TS with master...
I20260929 14:10:30.217991 4869 heartbeater.cc:511] Master 127.3.242.126:40359 requested a full tablet report, sending...
I20260929 14:10:30.219853 4593 ts_manager.cc:194] Registered new tserver with Master: dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:30.220342 4041 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017781s
I20260929 14:10:30.221491 4593 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44370
I20260929 14:10:30.222798 4041 test_util.cc:268] Using random seed: -1094563908
I20260929 14:10:30.248314 4593 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44380:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:10:30.251269 4593 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:10:30.336196 4760 tablet_service.cc:1467] Processing CreateTablet for tablet 97517573ba144283b9b7ec110600bba5 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260929 14:10:30.336792 4753 tablet_service.cc:1467] Processing CreateTablet for tablet f320a6a2097b4bf2927559b5797087ca (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260929 14:10:30.337654 4760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97517573ba144283b9b7ec110600bba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.338434 4753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f320a6a2097b4bf2927559b5797087ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.339463 4759 tablet_service.cc:1467] Processing CreateTablet for tablet 61659e7e13f94f06ade1d6a63d40c865 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260929 14:10:30.340734 4759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61659e7e13f94f06ade1d6a63d40c865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.342394 4758 tablet_service.cc:1467] Processing CreateTablet for tablet 4d6ebbf98eac48d392118aa29cdd0cb8 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260929 14:10:30.343655 4758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d6ebbf98eac48d392118aa29cdd0cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.344895 4754 tablet_service.cc:1467] Processing CreateTablet for tablet 57edb5a9c58445679fb36c0d352112e3 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260929 14:10:30.346176 4754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57edb5a9c58445679fb36c0d352112e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.346462 4757 tablet_service.cc:1467] Processing CreateTablet for tablet 4313956feee54744b19e2bf91abfa6c6 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260929 14:10:30.347702 4757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4313956feee54744b19e2bf91abfa6c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.350244 4756 tablet_service.cc:1467] Processing CreateTablet for tablet 0474cce108714cf3a6811d25f5ce09f4 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260929 14:10:30.351559 4756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0474cce108714cf3a6811d25f5ce09f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.351258 4752 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae676f8e6814c8296d465f3764c77e9 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260929 14:10:30.352474 4752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae676f8e6814c8296d465f3764c77e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.353598 4755 tablet_service.cc:1467] Processing CreateTablet for tablet b10fa99b33bb43399376921aab640714 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260929 14:10:30.354815 4755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b10fa99b33bb43399376921aab640714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.365276 4884 tablet_bootstrap.cc:492] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938: Bootstrap starting.
I20260929 14:10:30.371255 4884 tablet_bootstrap.cc:654] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.383361 4884 tablet_bootstrap.cc:492] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938: No bootstrap required, opened a new log
I20260929 14:10:30.383831 4884 ts_tablet_manager.cc:1397] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20260929 14:10:30.386216 4685 tablet_service.cc:1467] Processing CreateTablet for tablet 97517573ba144283b9b7ec110600bba5 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260929 14:10:30.386490 4884 raft_consensus.cc:348] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.387207 4884 raft_consensus.cc:374] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.387544 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97517573ba144283b9b7ec110600bba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.388864 4678 tablet_service.cc:1467] Processing CreateTablet for tablet f320a6a2097b4bf2927559b5797087ca (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260929 14:10:30.390161 4678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f320a6a2097b4bf2927559b5797087ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.394953 4681 tablet_service.cc:1467] Processing CreateTablet for tablet 0474cce108714cf3a6811d25f5ce09f4 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260929 14:10:30.396333 4681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0474cce108714cf3a6811d25f5ce09f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.398447 4682 tablet_service.cc:1467] Processing CreateTablet for tablet 4313956feee54744b19e2bf91abfa6c6 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260929 14:10:30.399817 4682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4313956feee54744b19e2bf91abfa6c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.387499 4884 raft_consensus.cc:729] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.401530 4683 tablet_service.cc:1467] Processing CreateTablet for tablet 4d6ebbf98eac48d392118aa29cdd0cb8 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260929 14:10:30.402089 4884 consensus_queue.cc:260] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.402808 4683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d6ebbf98eac48d392118aa29cdd0cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.406584 4684 tablet_service.cc:1467] Processing CreateTablet for tablet 61659e7e13f94f06ade1d6a63d40c865 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260929 14:10:30.407083 4826 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae676f8e6814c8296d465f3764c77e9 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260929 14:10:30.408620 4684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61659e7e13f94f06ade1d6a63d40c865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.411178 4827 tablet_service.cc:1467] Processing CreateTablet for tablet f320a6a2097b4bf2927559b5797087ca (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260929 14:10:30.414669 4828 tablet_service.cc:1467] Processing CreateTablet for tablet 57edb5a9c58445679fb36c0d352112e3 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260929 14:10:30.418879 4830 tablet_service.cc:1467] Processing CreateTablet for tablet 0474cce108714cf3a6811d25f5ce09f4 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260929 14:10:30.422710 4832 tablet_service.cc:1467] Processing CreateTablet for tablet 4d6ebbf98eac48d392118aa29cdd0cb8 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260929 14:10:30.425818 4833 tablet_service.cc:1467] Processing CreateTablet for tablet 61659e7e13f94f06ade1d6a63d40c865 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260929 14:10:30.433040 4834 tablet_service.cc:1467] Processing CreateTablet for tablet 97517573ba144283b9b7ec110600bba5 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260929 14:10:30.434540 4760 tablet_service.cc:1467] Processing CreateTablet for tablet 7d06d6f2848041fcbbe5f57ce706e118 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260929 14:10:30.434387 4825 tablet_service.cc:1467] Processing CreateTablet for tablet 7d06d6f2848041fcbbe5f57ce706e118 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260929 14:10:30.439426 4760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d06d6f2848041fcbbe5f57ce706e118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.435254 4679 tablet_service.cc:1467] Processing CreateTablet for tablet 57edb5a9c58445679fb36c0d352112e3 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260929 14:10:30.431610 4677 tablet_service.cc:1467] Processing CreateTablet for tablet 6ae676f8e6814c8296d465f3764c77e9 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260929 14:10:30.449072 4679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57edb5a9c58445679fb36c0d352112e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.449740 4677 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae676f8e6814c8296d465f3764c77e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.451604 4676 tablet_service.cc:1467] Processing CreateTablet for tablet 7d06d6f2848041fcbbe5f57ce706e118 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260929 14:10:30.452860 4676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d06d6f2848041fcbbe5f57ce706e118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.430090 4826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ae676f8e6814c8296d465f3764c77e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.435920 4829 tablet_service.cc:1467] Processing CreateTablet for tablet b10fa99b33bb43399376921aab640714 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260929 14:10:30.457600 4795 heartbeater.cc:503] Master 127.3.242.126:40359 was elected leader, sending a full tablet report...
I20260929 14:10:30.432003 4680 tablet_service.cc:1467] Processing CreateTablet for tablet b10fa99b33bb43399376921aab640714 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260929 14:10:30.470707 4680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b10fa99b33bb43399376921aab640714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.473671 4832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d6ebbf98eac48d392118aa29cdd0cb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.477658 4834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97517573ba144283b9b7ec110600bba5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.435824 4831 tablet_service.cc:1467] Processing CreateTablet for tablet 4313956feee54744b19e2bf91abfa6c6 (DEFAULT_TABLE table=test-workload [id=737b897f5d2d4fa3a2c70afe31df102e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260929 14:10:30.485597 4831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4313956feee54744b19e2bf91abfa6c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.486512 4829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b10fa99b33bb43399376921aab640714. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.489564 4827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f320a6a2097b4bf2927559b5797087ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.490504 4828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57edb5a9c58445679fb36c0d352112e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.491070 4884 ts_tablet_manager.cc:1428] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938: Time spent starting tablet: real 0.107s user 0.028s sys 0.024s
I20260929 14:10:30.492601 4884 tablet_bootstrap.cc:492] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938: Bootstrap starting.
I20260929 14:10:30.493572 4825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d06d6f2848041fcbbe5f57ce706e118. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.496884 4830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0474cce108714cf3a6811d25f5ce09f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.501585 4833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61659e7e13f94f06ade1d6a63d40c865. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:30.520495 4887 tablet_bootstrap.cc:492] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae: Bootstrap starting.
I20260929 14:10:30.531155 4884 tablet_bootstrap.cc:654] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.538936 4887 tablet_bootstrap.cc:654] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.579527 4884 tablet_bootstrap.cc:492] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938: No bootstrap required, opened a new log
I20260929 14:10:30.580036 4884 ts_tablet_manager.cc:1397] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938: Time spent bootstrapping tablet: real 0.088s user 0.023s sys 0.017s
I20260929 14:10:30.581037 4889 tablet_bootstrap.cc:492] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a: Bootstrap starting.
I20260929 14:10:30.582638 4884 raft_consensus.cc:348] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.585860 4884 raft_consensus.cc:374] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.586163 4884 raft_consensus.cc:729] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.587213 4887 tablet_bootstrap.cc:492] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae: No bootstrap required, opened a new log
I20260929 14:10:30.587843 4887 ts_tablet_manager.cc:1397] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.014s
I20260929 14:10:30.590487 4887 raft_consensus.cc:348] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.591189 4887 raft_consensus.cc:374] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.591478 4887 raft_consensus.cc:729] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.592136 4887 consensus_queue.cc:260] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.587347 4884 consensus_queue.cc:260] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.603775 4884 ts_tablet_manager.cc:1428] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260929 14:10:30.607635 4884 tablet_bootstrap.cc:492] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938: Bootstrap starting.
I20260929 14:10:30.610039 4886 raft_consensus.cc:482] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:30.610536 4886 raft_consensus.cc:504] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.612317 4720 heartbeater.cc:503] Master 127.3.242.126:40359 was elected leader, sending a full tablet report...
I20260929 14:10:30.613001 4887 ts_tablet_manager.cc:1428] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae: Time spent starting tablet: real 0.025s user 0.020s sys 0.001s
I20260929 14:10:30.615175 4887 tablet_bootstrap.cc:492] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae: Bootstrap starting.
I20260929 14:10:30.617254 4889 tablet_bootstrap.cc:654] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.623706 4886 leader_election.cc:288] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:30.626163 4889 tablet_bootstrap.cc:492] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a: No bootstrap required, opened a new log
I20260929 14:10:30.626789 4889 ts_tablet_manager.cc:1397] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a: Time spent bootstrapping tablet: real 0.046s user 0.011s sys 0.003s
I20260929 14:10:30.629070 4887 tablet_bootstrap.cc:654] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.630177 4889 raft_consensus.cc:348] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.631081 4889 raft_consensus.cc:374] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.623711 4884 tablet_bootstrap.cc:654] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.631511 4889 raft_consensus.cc:729] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.632675 4889 consensus_queue.cc:260] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.645152 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae" is_pre_election: true
I20260929 14:10:30.647320 4844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a" is_pre_election: true
I20260929 14:10:30.648434 4887 tablet_bootstrap.cc:492] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae: No bootstrap required, opened a new log
I20260929 14:10:30.648463 4869 heartbeater.cc:503] Master 127.3.242.126:40359 was elected leader, sending a full tablet report...
I20260929 14:10:30.648804 4891 raft_consensus.cc:482] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:30.649016 4887 ts_tablet_manager.cc:1397] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.011s
W20260929 14:10:30.649019 4733 leader_election.cc:341] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:30.649312 4891 raft_consensus.cc:504] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
W20260929 14:10:30.651467 4734 leader_election.cc:341] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:30.651893 4891 leader_election.cc:288] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:30.651799 4887 raft_consensus.cc:348] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.652582 4887 raft_consensus.cc:374] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.652900 4887 raft_consensus.cc:729] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.651893 4734 leader_election.cc:302] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 376e574c0386490bbc10b1c68119b938; no voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a
I20260929 14:10:30.653730 4887 consensus_queue.cc:260] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.654623 4886 raft_consensus.cc:2726] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:10:30.653733 4889 ts_tablet_manager.cc:1428] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a: Time spent starting tablet: real 0.027s user 0.015s sys 0.005s
I20260929 14:10:30.654953 4884 tablet_bootstrap.cc:492] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938: No bootstrap required, opened a new log
I20260929 14:10:30.655392 4884 ts_tablet_manager.cc:1397] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938: Time spent bootstrapping tablet: real 0.048s user 0.018s sys 0.001s
I20260929 14:10:30.655805 4889 tablet_bootstrap.cc:492] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a: Bootstrap starting.
I20260929 14:10:30.657928 4884 raft_consensus.cc:348] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.658712 4884 raft_consensus.cc:374] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.659034 4884 raft_consensus.cc:729] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.659727 4884 consensus_queue.cc:260] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.661906 4884 ts_tablet_manager.cc:1428] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260929 14:10:30.661978 4889 tablet_bootstrap.cc:654] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.663516 4887 ts_tablet_manager.cc:1428] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae: Time spent starting tablet: real 0.014s user 0.002s sys 0.004s
I20260929 14:10:30.664391 4887 tablet_bootstrap.cc:492] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae: Bootstrap starting.
I20260929 14:10:30.668047 4884 tablet_bootstrap.cc:492] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938: Bootstrap starting.
I20260929 14:10:30.671583 4887 tablet_bootstrap.cc:654] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.675891 4884 tablet_bootstrap.cc:654] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.684094 4891 raft_consensus.cc:482] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:30.684784 4891 raft_consensus.cc:504] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.687458 4891 leader_election.cc:288] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:30.702181 4889 tablet_bootstrap.cc:492] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a: No bootstrap required, opened a new log
I20260929 14:10:30.702850 4889 ts_tablet_manager.cc:1397] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a: Time spent bootstrapping tablet: real 0.047s user 0.024s sys 0.015s
I20260929 14:10:30.708391 4889 raft_consensus.cc:348] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.709151 4889 raft_consensus.cc:374] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.709487 4889 raft_consensus.cc:729] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.710170 4844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97517573ba144283b9b7ec110600bba5" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a" is_pre_election: true
I20260929 14:10:30.710523 4889 consensus_queue.cc:260] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.709597 4884 tablet_bootstrap.cc:492] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938: No bootstrap required, opened a new log
I20260929 14:10:30.711241 4843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61659e7e13f94f06ade1d6a63d40c865" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a" is_pre_election: true
I20260929 14:10:30.711524 4895 raft_consensus.cc:482] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:30.711719 4884 ts_tablet_manager.cc:1397] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938: Time spent bootstrapping tablet: real 0.044s user 0.020s sys 0.005s
I20260929 14:10:30.711966 4895 raft_consensus.cc:504] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.712527 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97517573ba144283b9b7ec110600bba5" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938" is_pre_election: true
I20260929 14:10:30.712937 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61659e7e13f94f06ade1d6a63d40c865" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938" is_pre_election: true
W20260929 14:10:30.712973 4658 leader_election.cc:341] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873): Illegal state: must be running to vote when last-logged opid is not known
W20260929 14:10:30.714571 4658 leader_election.cc:341] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601): Illegal state: must be running to vote when last-logged opid is not known
W20260929 14:10:30.715219 4658 leader_election.cc:341] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:30.715662 4658 leader_election.cc:302] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5898905d9cda47a88faa1bc424bad7ae; no voters: 376e574c0386490bbc10b1c68119b938, dab47308abbe40bcb7bb9eb70781192a
I20260929 14:10:30.716122 4844 raft_consensus.cc:2454] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 0.
I20260929 14:10:30.716580 4891 raft_consensus.cc:2726] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:10:30.716789 4887 tablet_bootstrap.cc:492] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae: No bootstrap required, opened a new log
I20260929 14:10:30.717419 4887 ts_tablet_manager.cc:1397] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae: Time spent bootstrapping tablet: real 0.053s user 0.020s sys 0.022s
I20260929 14:10:30.717005 4884 raft_consensus.cc:348] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.717911 4884 raft_consensus.cc:374] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.718209 4884 raft_consensus.cc:729] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.719017 4884 consensus_queue.cc:260] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.719944 4658 leader_election.cc:302] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a; no voters: 376e574c0386490bbc10b1c68119b938
I20260929 14:10:30.720093 4887 raft_consensus.cc:348] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.720808 4887 raft_consensus.cc:374] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.720813 4891 raft_consensus.cc:2781] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:30.721462 4891 raft_consensus.cc:482] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:30.721825 4891 raft_consensus.cc:3037] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:30.725653 4895 leader_election.cc:288] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523)
I20260929 14:10:30.721138 4887 raft_consensus.cc:729] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.729004 4887 consensus_queue.cc:260] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.738159 4884 ts_tablet_manager.cc:1428] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938: Time spent starting tablet: real 0.026s user 0.006s sys 0.001s
I20260929 14:10:30.739269 4884 tablet_bootstrap.cc:492] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938: Bootstrap starting.
I20260929 14:10:30.740103 4889 ts_tablet_manager.cc:1428] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a: Time spent starting tablet: real 0.036s user 0.007s sys 0.001s
I20260929 14:10:30.749262 4889 tablet_bootstrap.cc:492] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a: Bootstrap starting.
I20260929 14:10:30.754958 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97517573ba144283b9b7ec110600bba5" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938" is_pre_election: true
I20260929 14:10:30.755491 4889 tablet_bootstrap.cc:654] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a: Neither blocks nor log segments found. Creating new log.
W20260929 14:10:30.756673 4807 leader_election.cc:341] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:30.745033 4891 raft_consensus.cc:504] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.759310 4891 leader_election.cc:288] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 election: Requested vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:30.761586 4887 ts_tablet_manager.cc:1428] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae: Time spent starting tablet: real 0.044s user 0.012s sys 0.004s
I20260929 14:10:30.761197 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97517573ba144283b9b7ec110600bba5" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:30.762789 4887 tablet_bootstrap.cc:492] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae: Bootstrap starting.
I20260929 14:10:30.761600 4844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97517573ba144283b9b7ec110600bba5" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:30.763543 4844 raft_consensus.cc:3037] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:10:30.765535 4658 leader_election.cc:341] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:30.769896 4887 tablet_bootstrap.cc:654] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.772387 4844 raft_consensus.cc:2454] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 1.
I20260929 14:10:30.773970 4658 leader_election.cc:302] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a; no voters: 376e574c0386490bbc10b1c68119b938
I20260929 14:10:30.775266 4904 raft_consensus.cc:2781] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:30.775746 4904 raft_consensus.cc:686] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 1 LEADER]: Becoming Leader. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:30.776548 4904 consensus_queue.cc:237] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.778970 4884 tablet_bootstrap.cc:654] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.784842 4889 tablet_bootstrap.cc:492] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a: No bootstrap required, opened a new log
I20260929 14:10:30.785389 4889 ts_tablet_manager.cc:1397] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.015s
I20260929 14:10:30.785000 4592 catalog_manager.cc:5495] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65). New cstate: current_term: 1 leader_uuid: "5898905d9cda47a88faa1bc424bad7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:30.788204 4889 raft_consensus.cc:348] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.789000 4889 raft_consensus.cc:374] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.789337 4889 raft_consensus.cc:729] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.790061 4889 consensus_queue.cc:260] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.792120 4889 ts_tablet_manager.cc:1428] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260929 14:10:30.792986 4889 tablet_bootstrap.cc:492] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a: Bootstrap starting.
I20260929 14:10:30.797564 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97517573ba144283b9b7ec110600bba5" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae" is_pre_election: true
I20260929 14:10:30.799188 4808 leader_election.cc:302] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dab47308abbe40bcb7bb9eb70781192a; no voters: 376e574c0386490bbc10b1c68119b938, 5898905d9cda47a88faa1bc424bad7ae
I20260929 14:10:30.800397 4895 raft_consensus.cc:2726] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:10:30.800865 4889 tablet_bootstrap.cc:654] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.806102 4887 tablet_bootstrap.cc:492] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae: No bootstrap required, opened a new log
I20260929 14:10:30.806666 4887 ts_tablet_manager.cc:1397] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae: Time spent bootstrapping tablet: real 0.044s user 0.022s sys 0.020s
I20260929 14:10:30.809329 4887 raft_consensus.cc:348] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.809758 4884 tablet_bootstrap.cc:492] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938: No bootstrap required, opened a new log
I20260929 14:10:30.809907 4887 raft_consensus.cc:374] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.810109 4889 tablet_bootstrap.cc:492] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a: No bootstrap required, opened a new log
I20260929 14:10:30.810214 4884 ts_tablet_manager.cc:1397] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.009s
I20260929 14:10:30.810242 4887 raft_consensus.cc:729] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.810567 4889 ts_tablet_manager.cc:1397] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.007s
I20260929 14:10:30.811111 4887 consensus_queue.cc:260] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.813115 4887 ts_tablet_manager.cc:1428] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:10:30.812916 4884 raft_consensus.cc:348] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.813678 4884 raft_consensus.cc:374] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.813966 4884 raft_consensus.cc:729] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.814239 4887 tablet_bootstrap.cc:492] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae: Bootstrap starting.
I20260929 14:10:30.814262 4904 raft_consensus.cc:482] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:30.814801 4904 raft_consensus.cc:504] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.814697 4884 consensus_queue.cc:260] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.816845 4904 leader_election.cc:288] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:30.813117 4889 raft_consensus.cc:348] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.817631 4889 raft_consensus.cc:374] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.817934 4889 raft_consensus.cc:729] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.818562 4889 consensus_queue.cc:260] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.818994 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f320a6a2097b4bf2927559b5797087ca" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938" is_pre_election: true
I20260929 14:10:30.820286 4887 tablet_bootstrap.cc:654] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.820062 4844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f320a6a2097b4bf2927559b5797087ca" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a" is_pre_election: true
I20260929 14:10:30.821300 4889 ts_tablet_manager.cc:1428] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
W20260929 14:10:30.822232 4658 leader_election.cc:341] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:30.822785 4884 ts_tablet_manager.cc:1428] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
W20260929 14:10:30.822881 4658 leader_election.cc:341] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:30.823323 4658 leader_election.cc:302] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5898905d9cda47a88faa1bc424bad7ae; no voters: 376e574c0386490bbc10b1c68119b938, dab47308abbe40bcb7bb9eb70781192a
I20260929 14:10:30.823748 4884 tablet_bootstrap.cc:492] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938: Bootstrap starting.
I20260929 14:10:30.828635 4904 raft_consensus.cc:2726] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:10:30.829558 4884 tablet_bootstrap.cc:654] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.832576 4889 tablet_bootstrap.cc:492] T 6ae676f8e6814c8296d465f3764c77e9 P dab47308abbe40bcb7bb9eb70781192a: Bootstrap starting.
I20260929 14:10:30.841549 4889 tablet_bootstrap.cc:654] T 6ae676f8e6814c8296d465f3764c77e9 P dab47308abbe40bcb7bb9eb70781192a: Neither blocks nor log segments found. Creating new log.
W20260929 14:10:30.841848 4641 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260929 14:10:30.845332 4884 tablet_bootstrap.cc:492] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938: No bootstrap required, opened a new log
I20260929 14:10:30.845705 4887 tablet_bootstrap.cc:492] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae: No bootstrap required, opened a new log
I20260929 14:10:30.845928 4884 ts_tablet_manager.cc:1397] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.008s
I20260929 14:10:30.846230 4887 ts_tablet_manager.cc:1397] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.016s
I20260929 14:10:30.849331 4887 raft_consensus.cc:348] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.850196 4887 raft_consensus.cc:374] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.849769 4884 raft_consensus.cc:348] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.850569 4887 raft_consensus.cc:729] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.850754 4884 raft_consensus.cc:374] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.851167 4884 raft_consensus.cc:729] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.851553 4887 consensus_queue.cc:260] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.851792 4884 consensus_queue.cc:260] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.853538 4889 tablet_bootstrap.cc:492] T 6ae676f8e6814c8296d465f3764c77e9 P dab47308abbe40bcb7bb9eb70781192a: No bootstrap required, opened a new log
I20260929 14:10:30.853971 4889 ts_tablet_manager.cc:1397] T 6ae676f8e6814c8296d465f3764c77e9 P dab47308abbe40bcb7bb9eb70781192a: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.007s
I20260929 14:10:30.854326 4887 ts_tablet_manager.cc:1428] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260929 14:10:30.854609 4884 ts_tablet_manager.cc:1428] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20260929 14:10:30.855366 4887 tablet_bootstrap.cc:492] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae: Bootstrap starting.
I20260929 14:10:30.855482 4884 tablet_bootstrap.cc:492] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938: Bootstrap starting.
I20260929 14:10:30.856415 4889 raft_consensus.cc:348] T 6ae676f8e6814c8296d465f3764c77e9 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.857156 4889 raft_consensus.cc:374] T 6ae676f8e6814c8296d465f3764c77e9 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.857457 4889 raft_consensus.cc:729] T 6ae676f8e6814c8296d465f3764c77e9 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.858191 4889 consensus_queue.cc:260] T 6ae676f8e6814c8296d465f3764c77e9 P dab47308abbe40bcb7bb9eb70781192a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.860617 4889 ts_tablet_manager.cc:1428] T 6ae676f8e6814c8296d465f3764c77e9 P dab47308abbe40bcb7bb9eb70781192a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:10:30.861578 4889 tablet_bootstrap.cc:492] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a: Bootstrap starting.
I20260929 14:10:30.862419 4887 tablet_bootstrap.cc:654] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.863855 4884 tablet_bootstrap.cc:654] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.867673 4889 tablet_bootstrap.cc:654] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.881171 4884 tablet_bootstrap.cc:492] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938: No bootstrap required, opened a new log
I20260929 14:10:30.881815 4884 ts_tablet_manager.cc:1397] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.002s
I20260929 14:10:30.885241 4884 raft_consensus.cc:348] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.886241 4884 raft_consensus.cc:374] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.886600 4884 raft_consensus.cc:729] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.888633 4884 consensus_queue.cc:260] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.889866 4889 tablet_bootstrap.cc:492] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a: No bootstrap required, opened a new log
I20260929 14:10:30.890331 4889 ts_tablet_manager.cc:1397] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.013s
I20260929 14:10:30.891173 4884 ts_tablet_manager.cc:1428] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938: Time spent starting tablet: real 0.009s user 0.002s sys 0.005s
I20260929 14:10:30.892048 4884 tablet_bootstrap.cc:492] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938: Bootstrap starting.
I20260929 14:10:30.892808 4889 raft_consensus.cc:348] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.893543 4889 raft_consensus.cc:374] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.893817 4889 raft_consensus.cc:729] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.894490 4889 consensus_queue.cc:260] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.897024 4889 ts_tablet_manager.cc:1428] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260929 14:10:30.897900 4889 tablet_bootstrap.cc:492] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a: Bootstrap starting.
I20260929 14:10:30.898353 4884 tablet_bootstrap.cc:654] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.904531 4889 tablet_bootstrap.cc:654] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.905980 4895 raft_consensus.cc:482] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:30.906569 4895 raft_consensus.cc:504] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.909695 4884 tablet_bootstrap.cc:492] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938: No bootstrap required, opened a new log
I20260929 14:10:30.910133 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d06d6f2848041fcbbe5f57ce706e118" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938" is_pre_election: true
I20260929 14:10:30.910567 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d06d6f2848041fcbbe5f57ce706e118" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae" is_pre_election: true
I20260929 14:10:30.909029 4895 leader_election.cc:288] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601)
I20260929 14:10:30.910112 4884 ts_tablet_manager.cc:1397] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.008s
W20260929 14:10:30.911811 4807 leader_election.cc:341] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601): Illegal state: must be running to vote when last-logged opid is not known
W20260929 14:10:30.912716 4808 leader_election.cc:341] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:30.913100 4808 leader_election.cc:302] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dab47308abbe40bcb7bb9eb70781192a; no voters: 376e574c0386490bbc10b1c68119b938, 5898905d9cda47a88faa1bc424bad7ae
I20260929 14:10:30.913620 4884 raft_consensus.cc:348] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.914083 4895 raft_consensus.cc:2726] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:10:30.914297 4884 raft_consensus.cc:374] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.914561 4884 raft_consensus.cc:729] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.915357 4884 consensus_queue.cc:260] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.918221 4884 ts_tablet_manager.cc:1428] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:10:30.919178 4884 tablet_bootstrap.cc:492] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938: Bootstrap starting.
I20260929 14:10:30.921388 4895 raft_consensus.cc:482] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:30.921448 4886 raft_consensus.cc:482] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:30.921931 4895 raft_consensus.cc:504] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.922132 4886 raft_consensus.cc:504] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.924142 4895 leader_election.cc:288] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601)
I20260929 14:10:30.924922 4886 leader_election.cc:288] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:30.925513 4886 raft_consensus.cc:482] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:30.926127 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57edb5a9c58445679fb36c0d352112e3" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938" is_pre_election: true
I20260929 14:10:30.926295 4904 raft_consensus.cc:482] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:30.926755 4769 raft_consensus.cc:2454] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 0.
I20260929 14:10:30.926896 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57edb5a9c58445679fb36c0d352112e3" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae" is_pre_election: true
I20260929 14:10:30.926857 4904 raft_consensus.cc:504] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.927879 4807 leader_election.cc:302] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, dab47308abbe40bcb7bb9eb70781192a; no voters:
W20260929 14:10:30.928397 4808 leader_election.cc:341] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:30.928675 4895 raft_consensus.cc:2781] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:30.929028 4895 raft_consensus.cc:482] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:30.929360 4895 raft_consensus.cc:3037] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:30.929526 4889 tablet_bootstrap.cc:492] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a: No bootstrap required, opened a new log
I20260929 14:10:30.929930 4889 ts_tablet_manager.cc:1397] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.009s
I20260929 14:10:30.931591 4844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0474cce108714cf3a6811d25f5ce09f4" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a" is_pre_election: true
I20260929 14:10:30.932446 4844 raft_consensus.cc:2454] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 0.
I20260929 14:10:30.932654 4889 raft_consensus.cc:348] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.933609 4889 raft_consensus.cc:374] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:30.933604 4843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f320a6a2097b4bf2927559b5797087ca" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a" is_pre_election: true
I20260929 14:10:30.934123 4889 raft_consensus.cc:729] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Initialized, Role: FOLLOWER
I20260929 14:10:30.934943 4904 leader_election.cc:288] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:30.934890 4889 consensus_queue.cc:260] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
W20260929 14:10:30.936380 4733 leader_election.cc:341] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:30.937304 4658 leader_election.cc:302] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:30.940210 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f320a6a2097b4bf2927559b5797087ca" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae" is_pre_election: true
I20260929 14:10:30.941015 4695 raft_consensus.cc:2454] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 0.
I20260929 14:10:30.942629 4734 leader_election.cc:302] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, 5898905d9cda47a88faa1bc424bad7ae; no voters: dab47308abbe40bcb7bb9eb70781192a
I20260929 14:10:30.943387 4895 raft_consensus.cc:504] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.944077 4904 raft_consensus.cc:2781] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:30.944510 4904 raft_consensus.cc:482] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:30.944934 4904 raft_consensus.cc:3037] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:30.946092 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57edb5a9c58445679fb36c0d352112e3" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:30.946620 4769 raft_consensus.cc:3037] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:30.946537 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57edb5a9c58445679fb36c0d352112e3" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
W20260929 14:10:30.947886 4808 leader_election.cc:341] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:30.944720 4889 ts_tablet_manager.cc:1428] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a: Time spent starting tablet: real 0.015s user 0.004s sys 0.008s
I20260929 14:10:30.949127 4889 tablet_bootstrap.cc:492] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a: Bootstrap starting.
I20260929 14:10:30.926010 4886 raft_consensus.cc:504] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.953133 4843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b10fa99b33bb43399376921aab640714" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a" is_pre_election: true
I20260929 14:10:30.953474 4769 raft_consensus.cc:2454] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 1.
I20260929 14:10:30.953910 4843 raft_consensus.cc:2454] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 0.
I20260929 14:10:30.954485 4807 leader_election.cc:302] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, dab47308abbe40bcb7bb9eb70781192a; no voters: 5898905d9cda47a88faa1bc424bad7ae
I20260929 14:10:30.955368 4733 leader_election.cc:302] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:30.957357 4919 raft_consensus.cc:2781] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:30.958020 4895 leader_election.cc:288] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 election: Requested vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601)
I20260929 14:10:30.958117 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b10fa99b33bb43399376921aab640714" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae" is_pre_election: true
W20260929 14:10:30.959432 4734 leader_election.cc:341] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:30.959822 4918 raft_consensus.cc:2781] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:30.960050 4919 raft_consensus.cc:686] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 1 LEADER]: Becoming Leader. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Running, Role: LEADER
I20260929 14:10:30.960162 4918 raft_consensus.cc:482] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:30.960491 4918 raft_consensus.cc:3037] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:30.961761 4886 leader_election.cc:288] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:30.962996 4889 tablet_bootstrap.cc:654] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.966053 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0474cce108714cf3a6811d25f5ce09f4" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938" is_pre_election: true
I20260929 14:10:30.963045 4886 raft_consensus.cc:2781] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:30.966665 4769 raft_consensus.cc:2454] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 0.
I20260929 14:10:30.967511 4886 raft_consensus.cc:482] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:30.968494 4886 raft_consensus.cc:3037] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:30.970010 4918 raft_consensus.cc:504] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.960876 4919 consensus_queue.cc:237] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.972577 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f320a6a2097b4bf2927559b5797087ca" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
I20260929 14:10:30.973125 4695 raft_consensus.cc:3037] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:30.978123 4895 raft_consensus.cc:482] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:30.978721 4895 raft_consensus.cc:504] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.941008 4884 tablet_bootstrap.cc:654] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:30.982309 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae" is_pre_election: true
I20260929 14:10:30.983141 4694 raft_consensus.cc:2454] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 0.
I20260929 14:10:30.984560 4808 leader_election.cc:302] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:30.987679 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938" is_pre_election: true
I20260929 14:10:30.988476 4769 raft_consensus.cc:2454] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 0.
I20260929 14:10:30.990015 4895 leader_election.cc:288] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601)
I20260929 14:10:30.990669 4895 raft_consensus.cc:2781] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:30.991070 4895 raft_consensus.cc:482] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:30.991477 4895 raft_consensus.cc:3037] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:30.996450 4904 raft_consensus.cc:504] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:30.999470 4695 raft_consensus.cc:2454] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 1.
I20260929 14:10:31.011191 4734 leader_election.cc:302] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, 5898905d9cda47a88faa1bc424bad7ae; no voters:
I20260929 14:10:31.000408 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0474cce108714cf3a6811d25f5ce09f4" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:31.002422 4843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0474cce108714cf3a6811d25f5ce09f4" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:31.012801 4769 raft_consensus.cc:3037] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.012965 4843 raft_consensus.cc:3037] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.014344 4889 tablet_bootstrap.cc:492] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a: No bootstrap required, opened a new log
I20260929 14:10:31.014773 4889 ts_tablet_manager.cc:1397] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a: Time spent bootstrapping tablet: real 0.066s user 0.019s sys 0.024s
I20260929 14:10:31.017023 4889 raft_consensus.cc:348] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.017686 4889 raft_consensus.cc:374] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:31.017951 4889 raft_consensus.cc:729] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Initialized, Role: FOLLOWER
I20260929 14:10:31.018646 4889 consensus_queue.cc:260] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.019259 4769 raft_consensus.cc:2454] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 1.
I20260929 14:10:31.004057 4884 tablet_bootstrap.cc:492] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938: No bootstrap required, opened a new log
I20260929 14:10:31.009768 4918 leader_election.cc:288] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 election: Requested vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:31.020900 4884 ts_tablet_manager.cc:1397] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938: Time spent bootstrapping tablet: real 0.102s user 0.008s sys 0.003s
I20260929 14:10:31.020952 4889 ts_tablet_manager.cc:1428] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260929 14:10:31.021042 4918 raft_consensus.cc:2781] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:31.021715 4918 raft_consensus.cc:686] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Becoming Leader. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Running, Role: LEADER
I20260929 14:10:31.021978 4889 tablet_bootstrap.cc:492] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a: Bootstrap starting.
I20260929 14:10:31.022605 4918 consensus_queue.cc:237] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.023693 4884 raft_consensus.cc:348] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.023540 4592 catalog_manager.cc:5495] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a reported cstate change: term changed from 0 to 1, leader changed from <none> to dab47308abbe40bcb7bb9eb70781192a (127.3.242.67). New cstate: current_term: 1 leader_uuid: "dab47308abbe40bcb7bb9eb70781192a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: HEALTHY } } }
I20260929 14:10:31.002504 4904 leader_election.cc:288] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 election: Requested vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:31.000007 4920 raft_consensus.cc:482] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:31.025439 4920 raft_consensus.cc:504] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.024453 4884 raft_consensus.cc:374] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:31.026749 4884 raft_consensus.cc:729] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Initialized, Role: FOLLOWER
I20260929 14:10:31.027467 4920 leader_election.cc:288] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:31.009685 4844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f320a6a2097b4bf2927559b5797087ca" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:31.028986 4884 consensus_queue.cc:260] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.029822 4844 raft_consensus.cc:3037] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.006055 4887 tablet_bootstrap.cc:492] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae: No bootstrap required, opened a new log
I20260929 14:10:31.030324 4842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4313956feee54744b19e2bf91abfa6c6" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a" is_pre_election: true
I20260929 14:10:31.020336 4658 leader_election.cc:302] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, 5898905d9cda47a88faa1bc424bad7ae; no voters:
I20260929 14:10:31.030790 4887 ts_tablet_manager.cc:1397] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae: Time spent bootstrapping tablet: real 0.176s user 0.055s sys 0.088s
I20260929 14:10:31.031286 4904 raft_consensus.cc:2781] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:31.031739 4904 raft_consensus.cc:686] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 1 LEADER]: Becoming Leader. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:31.032524 4904 consensus_queue.cc:237] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.000890 4886 raft_consensus.cc:504] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.033810 4887 raft_consensus.cc:348] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.034951 4887 raft_consensus.cc:374] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:31.034519 4886 leader_election.cc:288] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 election: Requested vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:31.035624 4887 raft_consensus.cc:729] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Initialized, Role: FOLLOWER
I20260929 14:10:31.036473 4887 consensus_queue.cc:260] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.037781 4844 raft_consensus.cc:2454] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 1.
W20260929 14:10:31.038441 4733 leader_election.cc:341] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:31.039220 4886 raft_consensus.cc:482] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:31.039825 4887 ts_tablet_manager.cc:1428] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae: Time spent starting tablet: real 0.009s user 0.005s sys 0.003s
I20260929 14:10:31.039718 4886 raft_consensus.cc:504] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.041280 4887 tablet_bootstrap.cc:492] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae: Bootstrap starting.
I20260929 14:10:31.042371 4842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b10fa99b33bb43399376921aab640714" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:31.042896 4886 leader_election.cc:288] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:31.043146 4842 raft_consensus.cc:3037] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.043315 4841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61659e7e13f94f06ade1d6a63d40c865" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a" is_pre_election: true
W20260929 14:10:31.045456 4733 leader_election.cc:341] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:31.050632 4884 ts_tablet_manager.cc:1428] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938: Time spent starting tablet: real 0.029s user 0.006s sys 0.000s
I20260929 14:10:31.049789 4591 catalog_manager.cc:5495] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65). New cstate: current_term: 1 leader_uuid: "5898905d9cda47a88faa1bc424bad7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:31.051474 4884 tablet_bootstrap.cc:492] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938: Bootstrap starting.
I20260929 14:10:31.056236 4842 raft_consensus.cc:2454] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 1.
I20260929 14:10:31.052744 4843 raft_consensus.cc:2454] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 1.
I20260929 14:10:31.057305 4733 leader_election.cc:302] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:31.058092 4920 raft_consensus.cc:482] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:31.058521 4920 raft_consensus.cc:504] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.060372 4920 leader_election.cc:288] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:31.060832 4920 raft_consensus.cc:2781] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:31.061264 4920 raft_consensus.cc:686] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Becoming Leader. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Running, Role: LEADER
I20260929 14:10:31.042419 4895 raft_consensus.cc:504] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.065110 4895 leader_election.cc:288] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 election: Requested vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601)
I20260929 14:10:31.062047 4920 consensus_queue.cc:237] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.066237 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:31.067137 4769 raft_consensus.cc:3037] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.069118 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
I20260929 14:10:31.070258 4695 raft_consensus.cc:3037] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.073160 4884 tablet_bootstrap.cc:654] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:31.075865 4889 tablet_bootstrap.cc:654] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:31.064255 4843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97517573ba144283b9b7ec110600bba5" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a" is_pre_election: true
W20260929 14:10:31.080229 4734 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:35523, user_credentials={real_user=slave}} blocked reactor thread for 69171.8us
I20260929 14:10:31.082108 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4313956feee54744b19e2bf91abfa6c6" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae" is_pre_election: true
I20260929 14:10:31.079738 4843 raft_consensus.cc:2379] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 376e574c0386490bbc10b1c68119b938 in current term 1: Already voted for candidate 5898905d9cda47a88faa1bc424bad7ae in this term.
I20260929 14:10:31.086685 4887 tablet_bootstrap.cc:654] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:31.088483 4695 raft_consensus.cc:2454] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 1.
I20260929 14:10:31.088626 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b10fa99b33bb43399376921aab640714" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
I20260929 14:10:31.089330 4694 raft_consensus.cc:3037] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.089778 4808 leader_election.cc:302] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:31.086334 4769 raft_consensus.cc:2454] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 1.
W20260929 14:10:31.095103 4734 leader_election.cc:341] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:31.094527 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61659e7e13f94f06ade1d6a63d40c865" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae" is_pre_election: true
I20260929 14:10:31.095647 4895 raft_consensus.cc:2781] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:31.096251 4695 raft_consensus.cc:2454] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 0.
I20260929 14:10:31.095590 4734 leader_election.cc:302] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 376e574c0386490bbc10b1c68119b938; no voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a
I20260929 14:10:31.096938 4693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97517573ba144283b9b7ec110600bba5" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae" is_pre_election: true
I20260929 14:10:31.096400 4895 raft_consensus.cc:686] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 1 LEADER]: Becoming Leader. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Running, Role: LEADER
I20260929 14:10:31.097477 4925 raft_consensus.cc:2726] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:10:31.097646 4734 leader_election.cc:302] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, 5898905d9cda47a88faa1bc424bad7ae; no voters: dab47308abbe40bcb7bb9eb70781192a
I20260929 14:10:31.098547 4925 raft_consensus.cc:2781] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:31.098223 4895 consensus_queue.cc:237] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.099061 4734 leader_election.cc:302] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 376e574c0386490bbc10b1c68119b938; no voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a
I20260929 14:10:31.100895 4918 raft_consensus.cc:3037] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.098932 4925 raft_consensus.cc:482] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:31.107298 4925 raft_consensus.cc:3037] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.109325 4694 raft_consensus.cc:2454] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 1.
I20260929 14:10:31.112665 4918 raft_consensus.cc:2726] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:10:31.120780 4884 tablet_bootstrap.cc:492] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938: No bootstrap required, opened a new log
I20260929 14:10:31.121531 4884 ts_tablet_manager.cc:1397] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.016s
I20260929 14:10:31.116200 4925 raft_consensus.cc:504] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.125905 4884 raft_consensus.cc:348] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.126704 4884 raft_consensus.cc:374] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:31.126780 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61659e7e13f94f06ade1d6a63d40c865" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
I20260929 14:10:31.127039 4884 raft_consensus.cc:729] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Initialized, Role: FOLLOWER
I20260929 14:10:31.127561 4694 raft_consensus.cc:3037] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.127844 4884 consensus_queue.cc:260] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.130512 4884 ts_tablet_manager.cc:1428] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260929 14:10:31.129786 4591 catalog_manager.cc:5495] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a reported cstate change: term changed from 0 to 1, leader changed from <none> to dab47308abbe40bcb7bb9eb70781192a (127.3.242.67). New cstate: current_term: 1 leader_uuid: "dab47308abbe40bcb7bb9eb70781192a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: HEALTHY } } }
I20260929 14:10:31.133147 4889 tablet_bootstrap.cc:492] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a: No bootstrap required, opened a new log
I20260929 14:10:31.133877 4889 ts_tablet_manager.cc:1397] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a: Time spent bootstrapping tablet: real 0.112s user 0.011s sys 0.014s
I20260929 14:10:31.133378 4592 catalog_manager.cc:5495] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 reported cstate change: term changed from 0 to 1, leader changed from <none> to 376e574c0386490bbc10b1c68119b938 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "376e574c0386490bbc10b1c68119b938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:31.135525 4694 raft_consensus.cc:2454] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 1.
I20260929 14:10:31.135627 4592 catalog_manager.cc:5495] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 reported cstate change: term changed from 0 to 1, leader changed from <none> to 376e574c0386490bbc10b1c68119b938 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "376e574c0386490bbc10b1c68119b938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:31.137697 4843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61659e7e13f94f06ade1d6a63d40c865" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
W20260929 14:10:31.139132 4733 leader_election.cc:341] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:10:31.140049 4887 tablet_bootstrap.cc:492] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae: No bootstrap required, opened a new log
I20260929 14:10:31.136451 4889 raft_consensus.cc:348] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.141506 4889 raft_consensus.cc:374] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:31.141546 4887 ts_tablet_manager.cc:1397] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae: Time spent bootstrapping tablet: real 0.101s user 0.017s sys 0.002s
I20260929 14:10:31.141793 4889 raft_consensus.cc:729] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Initialized, Role: FOLLOWER
I20260929 14:10:31.142500 4889 consensus_queue.cc:260] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.137396 4925 leader_election.cc:288] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 election: Requested vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:31.144474 4734 leader_election.cc:302] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, 5898905d9cda47a88faa1bc424bad7ae; no voters: dab47308abbe40bcb7bb9eb70781192a
I20260929 14:10:31.144708 4887 raft_consensus.cc:348] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.145566 4887 raft_consensus.cc:374] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:31.145602 4925 raft_consensus.cc:2781] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:31.146360 4925 raft_consensus.cc:686] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Becoming Leader. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Running, Role: LEADER
I20260929 14:10:31.146049 4887 raft_consensus.cc:729] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Initialized, Role: FOLLOWER
I20260929 14:10:31.147439 4889 ts_tablet_manager.cc:1428] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a: Time spent starting tablet: real 0.013s user 0.004s sys 0.001s
I20260929 14:10:31.148277 4889 tablet_bootstrap.cc:492] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a: Bootstrap starting.
I20260929 14:10:31.147984 4887 consensus_queue.cc:260] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.148124 4925 consensus_queue.cc:237] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.150249 4887 ts_tablet_manager.cc:1428] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260929 14:10:31.151144 4887 tablet_bootstrap.cc:492] T 6ae676f8e6814c8296d465f3764c77e9 P 5898905d9cda47a88faa1bc424bad7ae: Bootstrap starting.
I20260929 14:10:31.153584 4889 tablet_bootstrap.cc:654] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:31.156911 4887 tablet_bootstrap.cc:654] T 6ae676f8e6814c8296d465f3764c77e9 P 5898905d9cda47a88faa1bc424bad7ae: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:31.158967 4889 tablet_bootstrap.cc:492] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a: No bootstrap required, opened a new log
I20260929 14:10:31.159252 4889 ts_tablet_manager.cc:1397] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 14:10:31.161198 4889 raft_consensus.cc:348] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.161921 4889 raft_consensus.cc:374] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:31.162168 4889 raft_consensus.cc:729] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Initialized, Role: FOLLOWER
I20260929 14:10:31.163213 4887 tablet_bootstrap.cc:492] T 6ae676f8e6814c8296d465f3764c77e9 P 5898905d9cda47a88faa1bc424bad7ae: No bootstrap required, opened a new log
I20260929 14:10:31.163676 4887 ts_tablet_manager.cc:1397] T 6ae676f8e6814c8296d465f3764c77e9 P 5898905d9cda47a88faa1bc424bad7ae: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.002s
I20260929 14:10:31.163465 4889 consensus_queue.cc:260] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.166319 4887 raft_consensus.cc:348] T 6ae676f8e6814c8296d465f3764c77e9 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.167013 4887 raft_consensus.cc:374] T 6ae676f8e6814c8296d465f3764c77e9 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:31.167269 4887 raft_consensus.cc:729] T 6ae676f8e6814c8296d465f3764c77e9 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Initialized, Role: FOLLOWER
I20260929 14:10:31.167883 4887 consensus_queue.cc:260] T 6ae676f8e6814c8296d465f3764c77e9 P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.169144 4889 ts_tablet_manager.cc:1428] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a: Time spent starting tablet: real 0.010s user 0.001s sys 0.005s
I20260929 14:10:31.169845 4887 ts_tablet_manager.cc:1428] T 6ae676f8e6814c8296d465f3764c77e9 P 5898905d9cda47a88faa1bc424bad7ae: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260929 14:10:31.171805 4887 tablet_bootstrap.cc:492] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae: Bootstrap starting.
I20260929 14:10:31.173463 4592 catalog_manager.cc:5495] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 reported cstate change: term changed from 0 to 1, leader changed from <none> to 376e574c0386490bbc10b1c68119b938 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "376e574c0386490bbc10b1c68119b938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:31.177018 4887 tablet_bootstrap.cc:654] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:31.183519 4887 tablet_bootstrap.cc:492] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae: No bootstrap required, opened a new log
I20260929 14:10:31.183908 4887 ts_tablet_manager.cc:1397] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 14:10:31.185667 4887 raft_consensus.cc:348] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.186164 4887 raft_consensus.cc:374] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:31.186412 4887 raft_consensus.cc:729] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Initialized, Role: FOLLOWER
I20260929 14:10:31.186932 4887 consensus_queue.cc:260] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.188534 4887 ts_tablet_manager.cc:1428] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 14:10:31.189558 4887 tablet_bootstrap.cc:492] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae: Bootstrap starting.
I20260929 14:10:31.191781 4925 raft_consensus.cc:482] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:31.192229 4925 raft_consensus.cc:504] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.194046 4887 tablet_bootstrap.cc:654] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:31.194535 4925 leader_election.cc:288] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:31.194870 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae676f8e6814c8296d465f3764c77e9" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae" is_pre_election: true
I20260929 14:10:31.195317 4843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae676f8e6814c8296d465f3764c77e9" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a" is_pre_election: true
I20260929 14:10:31.195611 4694 raft_consensus.cc:2454] T 6ae676f8e6814c8296d465f3764c77e9 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 0.
I20260929 14:10:31.195899 4843 raft_consensus.cc:2454] T 6ae676f8e6814c8296d465f3764c77e9 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 0.
I20260929 14:10:31.196732 4734 leader_election.cc:302] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, 5898905d9cda47a88faa1bc424bad7ae; no voters:
I20260929 14:10:31.197728 4925 raft_consensus.cc:2781] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:31.198100 4925 raft_consensus.cc:482] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:31.198442 4925 raft_consensus.cc:3037] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.200107 4887 tablet_bootstrap.cc:492] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae: No bootstrap required, opened a new log
I20260929 14:10:31.200595 4887 ts_tablet_manager.cc:1397] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260929 14:10:31.202872 4887 raft_consensus.cc:348] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.203655 4887 raft_consensus.cc:374] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:31.204008 4887 raft_consensus.cc:729] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Initialized, Role: FOLLOWER
I20260929 14:10:31.204596 4887 consensus_queue.cc:260] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.205255 4925 raft_consensus.cc:504] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.206518 4887 ts_tablet_manager.cc:1428] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260929 14:10:31.207736 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae676f8e6814c8296d465f3764c77e9" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
I20260929 14:10:31.208304 4694 raft_consensus.cc:3037] T 6ae676f8e6814c8296d465f3764c77e9 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.208395 4843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ae676f8e6814c8296d465f3764c77e9" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:31.208959 4843 raft_consensus.cc:3037] T 6ae676f8e6814c8296d465f3764c77e9 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.214437 4843 raft_consensus.cc:2454] T 6ae676f8e6814c8296d465f3764c77e9 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 1.
I20260929 14:10:31.214561 4925 leader_election.cc:288] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 election: Requested vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:31.215337 4694 raft_consensus.cc:2454] T 6ae676f8e6814c8296d465f3764c77e9 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 1.
I20260929 14:10:31.215476 4733 leader_election.cc:302] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:31.216794 4925 raft_consensus.cc:2781] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:31.217289 4925 raft_consensus.cc:686] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Becoming Leader. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Running, Role: LEADER
I20260929 14:10:31.218257 4925 consensus_queue.cc:237] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.221001 4904 raft_consensus.cc:482] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:31.221521 4904 raft_consensus.cc:504] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.223454 4904 leader_election.cc:288] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:31.224773 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d06d6f2848041fcbbe5f57ce706e118" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938" is_pre_election: true
I20260929 14:10:31.225008 4843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d06d6f2848041fcbbe5f57ce706e118" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a" is_pre_election: true
I20260929 14:10:31.225335 4769 raft_consensus.cc:2454] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 0.
I20260929 14:10:31.225672 4843 raft_consensus.cc:2454] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 0.
I20260929 14:10:31.226406 4658 leader_election.cc:302] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, 5898905d9cda47a88faa1bc424bad7ae; no voters:
I20260929 14:10:31.226166 4590 catalog_manager.cc:5495] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 reported cstate change: term changed from 0 to 1, leader changed from <none> to 376e574c0386490bbc10b1c68119b938 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "376e574c0386490bbc10b1c68119b938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:31.227113 4904 raft_consensus.cc:2781] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:31.227465 4904 raft_consensus.cc:482] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:31.227790 4904 raft_consensus.cc:3037] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.232295 4904 raft_consensus.cc:504] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.234115 4904 leader_election.cc:288] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 election: Requested vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:31.234925 4843 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d06d6f2848041fcbbe5f57ce706e118" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:31.235369 4843 raft_consensus.cc:3037] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.235261 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d06d6f2848041fcbbe5f57ce706e118" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:31.235860 4769 raft_consensus.cc:3037] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.239663 4843 raft_consensus.cc:2454] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 1.
I20260929 14:10:31.240221 4769 raft_consensus.cc:2454] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 1.
I20260929 14:10:31.240615 4658 leader_election.cc:302] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:31.241351 4904 raft_consensus.cc:2781] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:31.241734 4904 raft_consensus.cc:686] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 1 LEADER]: Becoming Leader. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:31.242460 4904 consensus_queue.cc:237] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.248901 4590 catalog_manager.cc:5495] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae reported cstate change: term changed from 0 to 1, leader changed from <none> to 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65). New cstate: current_term: 1 leader_uuid: "5898905d9cda47a88faa1bc424bad7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:31.254120 4904 consensus_queue.cc:1035] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:31.264995 4891 consensus_queue.cc:1035] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:31.287829 4895 raft_consensus.cc:482] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:31.288427 4895 raft_consensus.cc:504] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.290261 4895 leader_election.cc:288] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601)
I20260929 14:10:31.291069 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61659e7e13f94f06ade1d6a63d40c865" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae" is_pre_election: true
I20260929 14:10:31.291214 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61659e7e13f94f06ade1d6a63d40c865" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938" is_pre_election: true
I20260929 14:10:31.291965 4694 raft_consensus.cc:2379] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dab47308abbe40bcb7bb9eb70781192a in current term 1: Already voted for candidate 376e574c0386490bbc10b1c68119b938 in this term.
I20260929 14:10:31.292909 4808 leader_election.cc:302] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dab47308abbe40bcb7bb9eb70781192a; no voters: 376e574c0386490bbc10b1c68119b938, 5898905d9cda47a88faa1bc424bad7ae
I20260929 14:10:31.293668 4895 raft_consensus.cc:3037] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.297984 4895 raft_consensus.cc:2726] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:10:31.355329 4895 consensus_queue.cc:1035] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:31.368605 4694 raft_consensus.cc:3037] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.375029 4919 consensus_queue.cc:1035] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:31.405390 4919 raft_consensus.cc:482] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:10:31.405874 4919 raft_consensus.cc:504] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.407410 4919 leader_election.cc:288] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601)
I20260929 14:10:31.408247 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4313956feee54744b19e2bf91abfa6c6" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae" is_pre_election: true
I20260929 14:10:31.408358 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4313956feee54744b19e2bf91abfa6c6" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938" is_pre_election: true
I20260929 14:10:31.408805 4694 raft_consensus.cc:2454] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 0.
I20260929 14:10:31.408939 4769 raft_consensus.cc:2454] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 0.
I20260929 14:10:31.409742 4808 leader_election.cc:302] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:31.410511 4919 raft_consensus.cc:2781] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:10:31.410843 4919 raft_consensus.cc:482] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:10:31.411123 4919 raft_consensus.cc:3037] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.415931 4919 raft_consensus.cc:504] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.417778 4919 leader_election.cc:288] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 election: Requested vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601)
I20260929 14:10:31.418612 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4313956feee54744b19e2bf91abfa6c6" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
I20260929 14:10:31.418754 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4313956feee54744b19e2bf91abfa6c6" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:31.419083 4694 raft_consensus.cc:3037] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.419396 4769 raft_consensus.cc:3037] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:31.423336 4694 raft_consensus.cc:2454] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 1.
I20260929 14:10:31.424289 4808 leader_election.cc:302] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:31.424443 4769 raft_consensus.cc:2454] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 1.
I20260929 14:10:31.424950 4919 raft_consensus.cc:2781] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:31.425388 4919 raft_consensus.cc:686] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 1 LEADER]: Becoming Leader. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Running, Role: LEADER
I20260929 14:10:31.426208 4919 consensus_queue.cc:237] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.433194 4590 catalog_manager.cc:5495] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a reported cstate change: term changed from 0 to 1, leader changed from <none> to dab47308abbe40bcb7bb9eb70781192a (127.3.242.67). New cstate: current_term: 1 leader_uuid: "dab47308abbe40bcb7bb9eb70781192a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: HEALTHY } } }
I20260929 14:10:31.470115 4925 consensus_queue.cc:1035] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:31.480530 4920 consensus_queue.cc:1035] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:31.525319 4769 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "61659e7e13f94f06ade1d6a63d40c865"
dest_uuid: "376e574c0386490bbc10b1c68119b938"
mode: GRACEFUL
new_leader_uuid: "5898905d9cda47a88faa1bc424bad7ae"
from {username='slave'} at 127.0.0.1:56326
I20260929 14:10:31.525908 4769 raft_consensus.cc:595] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Received request to transfer leadership to TS 5898905d9cda47a88faa1bc424bad7ae
I20260929 14:10:31.536684 4904 consensus_queue.cc:1035] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:10:31.540943 4843 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8"
dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
mode: GRACEFUL
new_leader_uuid: "5898905d9cda47a88faa1bc424bad7ae"
from {username='slave'} at 127.0.0.1:44624
I20260929 14:10:31.541605 4843 raft_consensus.cc:595] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 1 LEADER]: Received request to transfer leadership to TS 5898905d9cda47a88faa1bc424bad7ae
I20260929 14:10:31.547848 4843 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4313956feee54744b19e2bf91abfa6c6"
dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
mode: GRACEFUL
new_leader_uuid: "5898905d9cda47a88faa1bc424bad7ae"
from {username='slave'} at 127.0.0.1:44624
I20260929 14:10:31.547693 4954 consensus_queue.cc:1035] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:10:31.548589 4843 raft_consensus.cc:595] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 1 LEADER]: Received request to transfer leadership to TS 5898905d9cda47a88faa1bc424bad7ae
I20260929 14:10:31.560421 4919 consensus_queue.cc:1035] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:31.567565 4923 raft_consensus.cc:979] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a: : Instructing follower 5898905d9cda47a88faa1bc424bad7ae to start an election
I20260929 14:10:31.568001 4895 raft_consensus.cc:1067] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 1 LEADER]: Signalling peer 5898905d9cda47a88faa1bc424bad7ae to start an election
I20260929 14:10:31.570744 4842 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57edb5a9c58445679fb36c0d352112e3"
dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
mode: GRACEFUL
new_leader_uuid: "376e574c0386490bbc10b1c68119b938"
from {username='slave'} at 127.0.0.1:44624
I20260929 14:10:31.571334 4842 raft_consensus.cc:595] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 1 LEADER]: Received request to transfer leadership to TS 376e574c0386490bbc10b1c68119b938
I20260929 14:10:31.571264 4695 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8"
dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
from {username='slave'} at 127.0.0.1:42746
I20260929 14:10:31.571977 4695 raft_consensus.cc:482] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:31.572335 4695 raft_consensus.cc:3037] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:31.573141 4895 consensus_queue.cc:1035] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:31.579053 4695 raft_consensus.cc:504] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.580174 4694 raft_consensus.cc:1226] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Rejecting Update request from peer dab47308abbe40bcb7bb9eb70781192a for earlier term 1. Current term is 2. Ops: []
I20260929 14:10:31.584024 4923 consensus_queue.cc:1046] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:31.584870 4895 raft_consensus.cc:3032] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:10:31.585206 4895 raft_consensus.cc:729] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Running, Role: LEADER
I20260929 14:10:31.585168 4920 consensus_queue.cc:1035] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:31.585901 4695 leader_election.cc:288] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 2 election: Requested vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:31.586159 4895 consensus_queue.cc:260] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.587404 4895 raft_consensus.cc:3037] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:31.588747 4842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:31.594049 4920 consensus_queue.cc:1035] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:31.594969 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:31.595599 4769 raft_consensus.cc:3037] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:31.610755 4842 raft_consensus.cc:2454] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 2.
I20260929 14:10:31.613289 4658 leader_election.cc:302] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:31.614270 4904 raft_consensus.cc:2781] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:10:31.614734 4904 raft_consensus.cc:686] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Becoming Leader. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:31.615772 4904 consensus_queue.cc:237] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.646644 4918 consensus_queue.cc:1035] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:10:31.626991 4920 consensus_queue.cc:1035] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.037s
I20260929 14:10:31.668490 4769 raft_consensus.cc:2454] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 2.
I20260929 14:10:31.678651 4918 raft_consensus.cc:979] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938: : Instructing follower 5898905d9cda47a88faa1bc424bad7ae to start an election
I20260929 14:10:31.679123 4918 raft_consensus.cc:1067] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Signalling peer 5898905d9cda47a88faa1bc424bad7ae to start an election
I20260929 14:10:31.682752 4694 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "61659e7e13f94f06ade1d6a63d40c865"
dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
from {username='slave'} at 127.0.0.1:42742
I20260929 14:10:31.683350 4694 raft_consensus.cc:482] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:31.683702 4694 raft_consensus.cc:3037] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:31.685588 4954 consensus_queue.cc:1035] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:10:31.685640 4918 consensus_queue.cc:1035] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:10:31.693925 4694 raft_consensus.cc:504] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.695264 4695 raft_consensus.cc:1226] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Rejecting Update request from peer 376e574c0386490bbc10b1c68119b938 for earlier term 1. Current term is 2. Ops: []
I20260929 14:10:31.695268 4593 catalog_manager.cc:5495] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae reported cstate change: term changed from 1 to 2, leader changed from dab47308abbe40bcb7bb9eb70781192a (127.3.242.67) to 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65). New cstate: current_term: 2 leader_uuid: "5898905d9cda47a88faa1bc424bad7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:31.706835 4694 leader_election.cc:288] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 2 election: Requested vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:31.710631 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61659e7e13f94f06ade1d6a63d40c865" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:31.711351 4769 raft_consensus.cc:3032] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:10:31.711745 4769 raft_consensus.cc:729] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Running, Role: LEADER
I20260929 14:10:31.715073 4694 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7d06d6f2848041fcbbe5f57ce706e118"
dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
mode: GRACEFUL
new_leader_uuid: "dab47308abbe40bcb7bb9eb70781192a"
from {username='slave'} at 127.0.0.1:34774
I20260929 14:10:31.715668 4694 raft_consensus.cc:595] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 1 LEADER]: Received request to transfer leadership to TS dab47308abbe40bcb7bb9eb70781192a
I20260929 14:10:31.712564 4769 consensus_queue.cc:260] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.717101 4769 raft_consensus.cc:3037] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:31.720983 4844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61659e7e13f94f06ade1d6a63d40c865" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:31.721761 4844 raft_consensus.cc:3037] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:31.722801 4769 raft_consensus.cc:2454] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 2.
I20260929 14:10:31.723783 4658 leader_election.cc:302] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, 5898905d9cda47a88faa1bc424bad7ae; no voters:
I20260929 14:10:31.724561 4904 raft_consensus.cc:2781] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:10:31.725023 4904 raft_consensus.cc:686] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Becoming Leader. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:31.725803 4904 consensus_queue.cc:237] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.736681 4954 consensus_queue.cc:1035] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:31.738183 4920 consensus_queue.cc:1035] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:31.739928 4844 raft_consensus.cc:2454] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 2.
I20260929 14:10:31.742094 4593 catalog_manager.cc:5495] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae reported cstate change: term changed from 1 to 2, leader changed from 376e574c0386490bbc10b1c68119b938 (127.3.242.66) to 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65). New cstate: current_term: 2 leader_uuid: "5898905d9cda47a88faa1bc424bad7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:31.748766 4954 raft_consensus.cc:979] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae: : Instructing follower dab47308abbe40bcb7bb9eb70781192a to start an election
I20260929 14:10:31.749239 4954 raft_consensus.cc:1067] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 1 LEADER]: Signalling peer dab47308abbe40bcb7bb9eb70781192a to start an election
I20260929 14:10:31.750687 4842 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7d06d6f2848041fcbbe5f57ce706e118"
dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
from {username='slave'} at 127.0.0.1:54058
I20260929 14:10:31.751206 4842 raft_consensus.cc:482] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:31.751796 4842 raft_consensus.cc:3037] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:31.758849 4842 raft_consensus.cc:504] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.763985 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d06d6f2848041fcbbe5f57ce706e118" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
I20260929 14:10:31.764567 4695 raft_consensus.cc:3032] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:10:31.764868 4695 raft_consensus.cc:729] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:31.765607 4695 consensus_queue.cc:260] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.766690 4695 raft_consensus.cc:3037] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:31.771054 4842 leader_election.cc:288] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 2 election: Requested vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601)
I20260929 14:10:31.772089 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d06d6f2848041fcbbe5f57ce706e118" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:31.772640 4770 raft_consensus.cc:3037] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:31.772851 4695 raft_consensus.cc:2454] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 2.
I20260929 14:10:31.773895 4808 leader_election.cc:302] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:31.774551 4919 raft_consensus.cc:2781] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:10:31.774988 4919 raft_consensus.cc:686] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [term 2 LEADER]: Becoming Leader. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Running, Role: LEADER
I20260929 14:10:31.775743 4919 consensus_queue.cc:237] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.778016 4770 raft_consensus.cc:2454] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 2.
I20260929 14:10:31.781761 4592 catalog_manager.cc:5495] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a reported cstate change: term changed from 1 to 2, leader changed from 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65) to dab47308abbe40bcb7bb9eb70781192a (127.3.242.67). New cstate: current_term: 2 leader_uuid: "dab47308abbe40bcb7bb9eb70781192a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: HEALTHY } } }
I20260929 14:10:31.957374 4923 raft_consensus.cc:979] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a: : Instructing follower 376e574c0386490bbc10b1c68119b938 to start an election
I20260929 14:10:31.957847 4919 raft_consensus.cc:1067] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 1 LEADER]: Signalling peer 376e574c0386490bbc10b1c68119b938 to start an election
I20260929 14:10:31.959256 4770 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57edb5a9c58445679fb36c0d352112e3"
dest_uuid: "376e574c0386490bbc10b1c68119b938"
from {username='slave'} at 127.0.0.1:37880
I20260929 14:10:31.959784 4770 raft_consensus.cc:482] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:31.960057 4770 raft_consensus.cc:3037] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:31.964545 4770 raft_consensus.cc:504] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.966089 4770 leader_election.cc:288] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 2 election: Requested vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:31.967144 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57edb5a9c58445679fb36c0d352112e3" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
I20260929 14:10:31.967206 4842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57edb5a9c58445679fb36c0d352112e3" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:31.967934 4842 raft_consensus.cc:3032] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:10:31.967901 4695 raft_consensus.cc:3037] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:31.968431 4842 raft_consensus.cc:729] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Running, Role: LEADER
I20260929 14:10:31.969257 4842 consensus_queue.cc:260] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.970479 4842 raft_consensus.cc:3037] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:31.974922 4695 raft_consensus.cc:2454] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 2.
I20260929 14:10:31.976002 4734 leader_election.cc:302] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, 5898905d9cda47a88faa1bc424bad7ae; no voters:
I20260929 14:10:31.976806 4966 raft_consensus.cc:2781] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:10:31.977025 4842 raft_consensus.cc:2454] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 2.
I20260929 14:10:31.977270 4966 raft_consensus.cc:686] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 2 LEADER]: Becoming Leader. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Running, Role: LEADER
I20260929 14:10:31.978173 4966 consensus_queue.cc:237] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:31.984297 4592 catalog_manager.cc:5495] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 reported cstate change: term changed from 1 to 2, leader changed from dab47308abbe40bcb7bb9eb70781192a (127.3.242.67) to 376e574c0386490bbc10b1c68119b938 (127.3.242.66). New cstate: current_term: 2 leader_uuid: "376e574c0386490bbc10b1c68119b938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:32.036126 4919 consensus_queue.cc:1035] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:32.037290 4923 consensus_queue.cc:1035] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:10:32.045217 4923 raft_consensus.cc:979] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a: : Instructing follower 5898905d9cda47a88faa1bc424bad7ae to start an election
I20260929 14:10:32.045636 4923 raft_consensus.cc:1067] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 1 LEADER]: Signalling peer 5898905d9cda47a88faa1bc424bad7ae to start an election
I20260929 14:10:32.047765 4694 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4313956feee54744b19e2bf91abfa6c6"
dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
from {username='slave'} at 127.0.0.1:42746
I20260929 14:10:32.050423 4694 raft_consensus.cc:482] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:32.050817 4694 raft_consensus.cc:3037] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:32.057427 4694 raft_consensus.cc:504] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:32.060959 4694 leader_election.cc:288] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 2 election: Requested vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:32.061990 4842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4313956feee54744b19e2bf91abfa6c6" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:32.062634 4842 raft_consensus.cc:3032] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:10:32.062965 4842 raft_consensus.cc:729] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Running, Role: LEADER
I20260929 14:10:32.063709 4842 consensus_queue.cc:260] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:32.064946 4842 raft_consensus.cc:3037] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:32.069315 4842 raft_consensus.cc:2454] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 2.
I20260929 14:10:32.061987 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4313956feee54744b19e2bf91abfa6c6" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:32.069965 4770 raft_consensus.cc:3037] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:32.070523 4658 leader_election.cc:302] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:32.071198 4904 raft_consensus.cc:2781] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:10:32.071666 4904 raft_consensus.cc:686] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Becoming Leader. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:32.072435 4904 consensus_queue.cc:237] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:32.075862 4770 raft_consensus.cc:2454] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 2.
I20260929 14:10:32.079051 4592 catalog_manager.cc:5495] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae reported cstate change: term changed from 1 to 2, leader changed from dab47308abbe40bcb7bb9eb70781192a (127.3.242.67) to 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65). New cstate: current_term: 2 leader_uuid: "5898905d9cda47a88faa1bc424bad7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:32.130872 4844 raft_consensus.cc:1261] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Refusing update from remote peer 5898905d9cda47a88faa1bc424bad7ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:32.132248 4967 consensus_queue.cc:1035] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:32.141552 4770 raft_consensus.cc:1261] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Refusing update from remote peer 5898905d9cda47a88faa1bc424bad7ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:32.142762 4967 consensus_queue.cc:1035] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:32.263402 4694 raft_consensus.cc:1261] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Refusing update from remote peer dab47308abbe40bcb7bb9eb70781192a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:32.264664 4923 consensus_queue.cc:1035] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:32.272138 4770 raft_consensus.cc:1261] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Refusing update from remote peer dab47308abbe40bcb7bb9eb70781192a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:32.274688 4923 consensus_queue.cc:1035] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:32.291915 4844 raft_consensus.cc:1261] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Refusing update from remote peer 5898905d9cda47a88faa1bc424bad7ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:32.293342 4967 consensus_queue.cc:1035] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:10:32.304667 4770 raft_consensus.cc:1261] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Refusing update from remote peer 5898905d9cda47a88faa1bc424bad7ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:32.306648 4972 consensus_queue.cc:1035] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:32.548326 4844 raft_consensus.cc:1261] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Refusing update from remote peer 376e574c0386490bbc10b1c68119b938: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:32.549539 4925 consensus_queue.cc:1035] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:32.557785 4694 raft_consensus.cc:1261] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Refusing update from remote peer 376e574c0386490bbc10b1c68119b938: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:32.559301 4966 consensus_queue.cc:1035] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:32.571117 4844 raft_consensus.cc:1261] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Refusing update from remote peer 5898905d9cda47a88faa1bc424bad7ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:32.572384 4972 consensus_queue.cc:1035] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:32.582006 4770 raft_consensus.cc:1261] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Refusing update from remote peer 5898905d9cda47a88faa1bc424bad7ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:32.583461 4972 consensus_queue.cc:1035] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:34.744369 4041 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260929 14:10:34.744980 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:34.769280 4694 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0474cce108714cf3a6811d25f5ce09f4"
dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
mode: GRACEFUL
new_leader_uuid: "dab47308abbe40bcb7bb9eb70781192a"
from {username='slave'} at 127.0.0.1:34782
I20260929 14:10:34.769798 4694 raft_consensus.cc:595] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 1 LEADER]: Received request to transfer leadership to TS dab47308abbe40bcb7bb9eb70781192a
I20260929 14:10:34.771888 4694 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8"
dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
mode: GRACEFUL
new_leader_uuid: "dab47308abbe40bcb7bb9eb70781192a"
from {username='slave'} at 127.0.0.1:34782
I20260929 14:10:34.772338 4694 raft_consensus.cc:595] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Received request to transfer leadership to TS dab47308abbe40bcb7bb9eb70781192a
I20260929 14:10:34.773171 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260929 14:10:34.773589 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:34.917173 4981 raft_consensus.cc:979] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae: : Instructing follower dab47308abbe40bcb7bb9eb70781192a to start an election
I20260929 14:10:34.917659 4977 raft_consensus.cc:1067] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Signalling peer dab47308abbe40bcb7bb9eb70781192a to start an election
I20260929 14:10:34.918903 4844 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8"
dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
from {username='slave'} at 127.0.0.1:54058
I20260929 14:10:34.919351 4844 raft_consensus.cc:482] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:34.919641 4844 raft_consensus.cc:3037] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:34.924649 4844 raft_consensus.cc:504] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:34.926584 4844 leader_election.cc:288] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 3 election: Requested vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601)
I20260929 14:10:34.927496 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:34.927433 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
I20260929 14:10:34.928122 4770 raft_consensus.cc:3037] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:34.928160 4694 raft_consensus.cc:3032] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:10:34.928582 4694 raft_consensus.cc:729] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:34.929345 4694 consensus_queue.cc:260] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:34.930526 4694 raft_consensus.cc:3037] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:34.934612 4770 raft_consensus.cc:2454] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 3.
I20260929 14:10:34.935817 4807 leader_election.cc:302] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:34.936569 4982 raft_consensus.cc:2781] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:10:34.936933 4694 raft_consensus.cc:2454] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 3.
I20260929 14:10:34.937767 4982 raft_consensus.cc:686] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 3 LEADER]: Becoming Leader. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Running, Role: LEADER
I20260929 14:10:34.938539 4982 consensus_queue.cc:237] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:34.945689 4593 catalog_manager.cc:5495] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a reported cstate change: term changed from 2 to 3, leader changed from 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65) to dab47308abbe40bcb7bb9eb70781192a (127.3.242.67). New cstate: current_term: 3 leader_uuid: "dab47308abbe40bcb7bb9eb70781192a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: HEALTHY } } }
I20260929 14:10:35.196117 4981 raft_consensus.cc:979] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae: : Instructing follower dab47308abbe40bcb7bb9eb70781192a to start an election
I20260929 14:10:35.196655 4977 raft_consensus.cc:1067] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 1 LEADER]: Signalling peer dab47308abbe40bcb7bb9eb70781192a to start an election
I20260929 14:10:35.198030 4844 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0474cce108714cf3a6811d25f5ce09f4"
dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
from {username='slave'} at 127.0.0.1:54058
I20260929 14:10:35.198566 4844 raft_consensus.cc:482] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:35.198876 4844 raft_consensus.cc:3037] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:35.203588 4844 raft_consensus.cc:504] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:35.205395 4844 leader_election.cc:288] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 2 election: Requested vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601)
I20260929 14:10:35.206313 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0474cce108714cf3a6811d25f5ce09f4" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
I20260929 14:10:35.206620 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0474cce108714cf3a6811d25f5ce09f4" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:35.206964 4694 raft_consensus.cc:3032] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:10:35.207151 4770 raft_consensus.cc:3037] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:35.207299 4694 raft_consensus.cc:729] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:35.208087 4694 consensus_queue.cc:260] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:35.209336 4694 raft_consensus.cc:3037] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:35.214069 4770 raft_consensus.cc:2454] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 2.
I20260929 14:10:35.215195 4807 leader_election.cc:302] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:35.215956 4694 raft_consensus.cc:2454] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 2.
I20260929 14:10:35.216003 4982 raft_consensus.cc:2781] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:10:35.216619 4982 raft_consensus.cc:686] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 2 LEADER]: Becoming Leader. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Running, Role: LEADER
I20260929 14:10:35.217521 4982 consensus_queue.cc:237] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:35.227267 4593 catalog_manager.cc:5495] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a reported cstate change: term changed from 1 to 2, leader changed from 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65) to dab47308abbe40bcb7bb9eb70781192a (127.3.242.67). New cstate: current_term: 2 leader_uuid: "dab47308abbe40bcb7bb9eb70781192a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: HEALTHY } } }
I20260929 14:10:35.404085 4770 raft_consensus.cc:1261] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Refusing update from remote peer dab47308abbe40bcb7bb9eb70781192a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:35.405267 4982 consensus_queue.cc:1035] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:35.416874 4694 raft_consensus.cc:1261] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Refusing update from remote peer dab47308abbe40bcb7bb9eb70781192a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:35.418275 4983 consensus_queue.cc:1035] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:35.645754 4694 raft_consensus.cc:1261] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Refusing update from remote peer dab47308abbe40bcb7bb9eb70781192a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:35.646991 4983 consensus_queue.cc:1035] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:35.655385 4770 raft_consensus.cc:1261] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Refusing update from remote peer dab47308abbe40bcb7bb9eb70781192a: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:35.656778 4983 consensus_queue.cc:1035] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:35.774356 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:35.779444 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:35.779888 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:36.780707 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:36.785763 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:36.786159 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:37.786983 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:37.792873 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:37.793355 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:38.794081 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:38.799552 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:38.799937 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:39.800894 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:39.806239 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:39.806639 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:40.807438 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:40.812875 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:40.813372 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:41.814121 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:41.819083 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:41.819516 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:42.820300 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:42.826304 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:42.826730 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:43.827600 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:43.833892 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:43.834225 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:44.862751 4041 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260929 14:10:44.871229 4842 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8"
dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
mode: GRACEFUL
new_leader_uuid: "5898905d9cda47a88faa1bc424bad7ae"
from {username='slave'} at 127.0.0.1:44624
I20260929 14:10:44.871930 4842 raft_consensus.cc:595] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 3 LEADER]: Received request to transfer leadership to TS 5898905d9cda47a88faa1bc424bad7ae
I20260929 14:10:44.878212 4842 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0474cce108714cf3a6811d25f5ce09f4"
dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
mode: GRACEFUL
new_leader_uuid: "5898905d9cda47a88faa1bc424bad7ae"
from {username='slave'} at 127.0.0.1:44624
I20260929 14:10:44.878744 4842 raft_consensus.cc:595] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 2 LEADER]: Received request to transfer leadership to TS 5898905d9cda47a88faa1bc424bad7ae
I20260929 14:10:44.882795 4769 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b10fa99b33bb43399376921aab640714"
dest_uuid: "376e574c0386490bbc10b1c68119b938"
mode: GRACEFUL
new_leader_uuid: "5898905d9cda47a88faa1bc424bad7ae"
from {username='slave'} at 127.0.0.1:56326
I20260929 14:10:44.883240 4769 raft_consensus.cc:595] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Received request to transfer leadership to TS 5898905d9cda47a88faa1bc424bad7ae
I20260929 14:10:44.887164 4769 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57edb5a9c58445679fb36c0d352112e3"
dest_uuid: "376e574c0386490bbc10b1c68119b938"
mode: GRACEFUL
new_leader_uuid: "5898905d9cda47a88faa1bc424bad7ae"
from {username='slave'} at 127.0.0.1:56326
I20260929 14:10:44.887568 4769 raft_consensus.cc:595] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 2 LEADER]: Received request to transfer leadership to TS 5898905d9cda47a88faa1bc424bad7ae
I20260929 14:10:44.891546 4769 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f320a6a2097b4bf2927559b5797087ca"
dest_uuid: "376e574c0386490bbc10b1c68119b938"
mode: GRACEFUL
new_leader_uuid: "5898905d9cda47a88faa1bc424bad7ae"
from {username='slave'} at 127.0.0.1:56326
I20260929 14:10:44.892001 4769 raft_consensus.cc:595] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Received request to transfer leadership to TS 5898905d9cda47a88faa1bc424bad7ae
I20260929 14:10:44.937829 5002 raft_consensus.cc:979] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938: : Instructing follower 5898905d9cda47a88faa1bc424bad7ae to start an election
I20260929 14:10:44.938314 5009 raft_consensus.cc:1067] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 2 LEADER]: Signalling peer 5898905d9cda47a88faa1bc424bad7ae to start an election
I20260929 14:10:44.939950 4694 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57edb5a9c58445679fb36c0d352112e3"
dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
from {username='slave'} at 127.0.0.1:42742
I20260929 14:10:44.940454 4694 raft_consensus.cc:482] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:44.940778 4694 raft_consensus.cc:3037] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:44.945734 4694 raft_consensus.cc:504] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:44.947244 4694 leader_election.cc:288] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 3 election: Requested vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:44.948204 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57edb5a9c58445679fb36c0d352112e3" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:44.948341 4842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57edb5a9c58445679fb36c0d352112e3" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:44.948978 4842 raft_consensus.cc:3037] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:44.948972 4769 raft_consensus.cc:3032] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:10:44.949461 4769 raft_consensus.cc:729] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Running, Role: LEADER
I20260929 14:10:44.950450 4769 consensus_queue.cc:260] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:44.951771 4769 raft_consensus.cc:3037] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:44.956705 4842 raft_consensus.cc:2454] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 3.
I20260929 14:10:44.957942 4658 leader_election.cc:302] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:44.958371 4769 raft_consensus.cc:2454] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 3.
I20260929 14:10:44.958712 5008 raft_consensus.cc:2781] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:10:44.959174 5008 raft_consensus.cc:686] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 3 LEADER]: Becoming Leader. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:44.960042 5008 consensus_queue.cc:237] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:44.966465 4592 catalog_manager.cc:5495] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae reported cstate change: term changed from 2 to 3, leader changed from 376e574c0386490bbc10b1c68119b938 (127.3.242.66) to 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65). New cstate: current_term: 3 leader_uuid: "5898905d9cda47a88faa1bc424bad7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:45.040472 5015 raft_consensus.cc:979] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a: : Instructing follower 5898905d9cda47a88faa1bc424bad7ae to start an election
I20260929 14:10:45.041071 5007 raft_consensus.cc:1067] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 2 LEADER]: Signalling peer 5898905d9cda47a88faa1bc424bad7ae to start an election
I20260929 14:10:45.042491 4694 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0474cce108714cf3a6811d25f5ce09f4"
dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
from {username='slave'} at 127.0.0.1:42746
I20260929 14:10:45.042995 4694 raft_consensus.cc:482] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:45.043277 4694 raft_consensus.cc:3037] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:45.047418 4694 raft_consensus.cc:504] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:45.049057 4694 leader_election.cc:288] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 3 election: Requested vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:45.049965 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0474cce108714cf3a6811d25f5ce09f4" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:45.050093 4842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0474cce108714cf3a6811d25f5ce09f4" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:45.050452 4769 raft_consensus.cc:3037] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:45.050606 4842 raft_consensus.cc:3032] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:10:45.050942 4842 raft_consensus.cc:729] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Running, Role: LEADER
I20260929 14:10:45.051589 4842 consensus_queue.cc:260] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:45.052513 4842 raft_consensus.cc:3037] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:45.055086 4769 raft_consensus.cc:2454] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 3.
I20260929 14:10:45.056241 4658 leader_election.cc:302] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, 5898905d9cda47a88faa1bc424bad7ae; no voters:
I20260929 14:10:45.056931 5008 raft_consensus.cc:2781] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:10:45.057403 5008 raft_consensus.cc:686] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 3 LEADER]: Becoming Leader. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:45.057981 4842 raft_consensus.cc:2454] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 3.
I20260929 14:10:45.058145 5008 consensus_queue.cc:237] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:45.059412 5002 raft_consensus.cc:979] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938: : Instructing follower 5898905d9cda47a88faa1bc424bad7ae to start an election
I20260929 14:10:45.059856 5002 raft_consensus.cc:1067] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Signalling peer 5898905d9cda47a88faa1bc424bad7ae to start an election
I20260929 14:10:45.061172 4694 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b10fa99b33bb43399376921aab640714"
dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
from {username='slave'} at 127.0.0.1:42742
I20260929 14:10:45.061620 4694 raft_consensus.cc:482] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:45.061897 4694 raft_consensus.cc:3037] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:45.066226 4592 catalog_manager.cc:5495] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae reported cstate change: term changed from 2 to 3, leader changed from dab47308abbe40bcb7bb9eb70781192a (127.3.242.67) to 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65). New cstate: current_term: 3 leader_uuid: "5898905d9cda47a88faa1bc424bad7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:45.066859 4694 raft_consensus.cc:504] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:45.068531 4694 leader_election.cc:288] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 2 election: Requested vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:45.069321 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b10fa99b33bb43399376921aab640714" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:45.069772 4769 raft_consensus.cc:3032] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:10:45.070055 4769 raft_consensus.cc:729] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Running, Role: LEADER
I20260929 14:10:45.070076 4842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b10fa99b33bb43399376921aab640714" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:45.070689 4842 raft_consensus.cc:3037] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:45.070992 4769 consensus_queue.cc:260] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:45.072247 4769 raft_consensus.cc:3037] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:45.077176 4842 raft_consensus.cc:2454] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 2.
I20260929 14:10:45.078282 4769 raft_consensus.cc:2454] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 2.
I20260929 14:10:45.078320 4658 leader_election.cc:302] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:45.079080 5008 raft_consensus.cc:2781] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:10:45.079515 5008 raft_consensus.cc:686] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Becoming Leader. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:45.080189 5008 consensus_queue.cc:237] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:45.086153 4592 catalog_manager.cc:5495] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae reported cstate change: term changed from 1 to 2, leader changed from 376e574c0386490bbc10b1c68119b938 (127.3.242.66) to 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65). New cstate: current_term: 2 leader_uuid: "5898905d9cda47a88faa1bc424bad7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:45.164676 5015 raft_consensus.cc:979] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a: : Instructing follower 5898905d9cda47a88faa1bc424bad7ae to start an election
I20260929 14:10:45.165091 5007 raft_consensus.cc:1067] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 3 LEADER]: Signalling peer 5898905d9cda47a88faa1bc424bad7ae to start an election
I20260929 14:10:45.166406 4694 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8"
dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
from {username='slave'} at 127.0.0.1:42746
I20260929 14:10:45.166857 4694 raft_consensus.cc:482] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:45.167125 4694 raft_consensus.cc:3037] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:45.171775 4694 raft_consensus.cc:504] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:45.173265 4694 leader_election.cc:288] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 4 election: Requested vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:45.173889 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:45.174211 4842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d6ebbf98eac48d392118aa29cdd0cb8" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:45.174430 4769 raft_consensus.cc:3037] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:45.174741 4842 raft_consensus.cc:3032] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:10:45.175004 4842 raft_consensus.cc:729] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Running, Role: LEADER
I20260929 14:10:45.175644 4842 consensus_queue.cc:260] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:45.176627 4842 raft_consensus.cc:3037] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:45.178933 4769 raft_consensus.cc:2454] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 4.
I20260929 14:10:45.179768 4658 leader_election.cc:302] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, 5898905d9cda47a88faa1bc424bad7ae; no voters:
I20260929 14:10:45.180397 5008 raft_consensus.cc:2781] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:10:45.180859 5008 raft_consensus.cc:686] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 4 LEADER]: Becoming Leader. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:45.181905 4842 raft_consensus.cc:2454] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 4.
I20260929 14:10:45.181825 5008 consensus_queue.cc:237] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:45.188392 4592 catalog_manager.cc:5495] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae reported cstate change: term changed from 3 to 4, leader changed from dab47308abbe40bcb7bb9eb70781192a (127.3.242.67) to 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65). New cstate: current_term: 4 leader_uuid: "5898905d9cda47a88faa1bc424bad7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:45.254594 5009 raft_consensus.cc:979] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938: : Instructing follower 5898905d9cda47a88faa1bc424bad7ae to start an election
I20260929 14:10:45.255069 5009 raft_consensus.cc:1067] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Signalling peer 5898905d9cda47a88faa1bc424bad7ae to start an election
I20260929 14:10:45.256347 4694 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f320a6a2097b4bf2927559b5797087ca"
dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
from {username='slave'} at 127.0.0.1:42742
I20260929 14:10:45.256703 4694 raft_consensus.cc:482] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:45.256910 4694 raft_consensus.cc:3037] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:45.260612 4694 raft_consensus.cc:504] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:45.261848 4694 leader_election.cc:288] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 2 election: Requested vote from peers 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:45.262624 4769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f320a6a2097b4bf2927559b5797087ca" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:45.262902 4842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f320a6a2097b4bf2927559b5797087ca" candidate_uuid: "5898905d9cda47a88faa1bc424bad7ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:45.263077 4769 raft_consensus.cc:3032] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:10:45.263388 4769 raft_consensus.cc:729] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Running, Role: LEADER
I20260929 14:10:45.263407 4842 raft_consensus.cc:3037] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:45.264021 4769 consensus_queue.cc:260] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:45.264873 4769 raft_consensus.cc:3037] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:45.267747 4842 raft_consensus.cc:2454] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 2.
I20260929 14:10:45.268555 4658 leader_election.cc:302] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5898905d9cda47a88faa1bc424bad7ae, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:45.268664 4769 raft_consensus.cc:2454] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5898905d9cda47a88faa1bc424bad7ae in term 2.
I20260929 14:10:45.269160 5008 raft_consensus.cc:2781] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:10:45.269635 5008 raft_consensus.cc:686] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Becoming Leader. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:45.270387 5008 consensus_queue.cc:237] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:45.277108 4592 catalog_manager.cc:5495] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae reported cstate change: term changed from 1 to 2, leader changed from 376e574c0386490bbc10b1c68119b938 (127.3.242.66) to 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65). New cstate: current_term: 2 leader_uuid: "5898905d9cda47a88faa1bc424bad7ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:45.429476 4842 raft_consensus.cc:1261] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Refusing update from remote peer 5898905d9cda47a88faa1bc424bad7ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:45.430531 5008 consensus_queue.cc:1035] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:45.440587 4769 raft_consensus.cc:1261] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Refusing update from remote peer 5898905d9cda47a88faa1bc424bad7ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:45.441932 5008 consensus_queue.cc:1035] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:45.497743 4770 raft_consensus.cc:1261] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Refusing update from remote peer 5898905d9cda47a88faa1bc424bad7ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:45.498996 5003 consensus_queue.cc:1035] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:45.507939 4842 raft_consensus.cc:1261] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Refusing update from remote peer 5898905d9cda47a88faa1bc424bad7ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:45.509438 5003 consensus_queue.cc:1035] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:45.567251 4770 raft_consensus.cc:1261] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 4 FOLLOWER]: Refusing update from remote peer 5898905d9cda47a88faa1bc424bad7ae: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260929 14:10:45.568333 5003 consensus_queue.cc:1035] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:10:45.577864 4842 raft_consensus.cc:1261] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 4 FOLLOWER]: Refusing update from remote peer 5898905d9cda47a88faa1bc424bad7ae: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260929 14:10:45.579170 5022 consensus_queue.cc:1035] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:10:45.601102 4770 raft_consensus.cc:1261] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Refusing update from remote peer 5898905d9cda47a88faa1bc424bad7ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:45.602262 5022 consensus_queue.cc:1035] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:45.606845 4842 raft_consensus.cc:1261] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Refusing update from remote peer 5898905d9cda47a88faa1bc424bad7ae: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:45.608090 5022 consensus_queue.cc:1035] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:45.652945 4770 raft_consensus.cc:1261] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Refusing update from remote peer 5898905d9cda47a88faa1bc424bad7ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:45.654060 5008 consensus_queue.cc:1035] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:45.661839 4842 raft_consensus.cc:1261] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Refusing update from remote peer 5898905d9cda47a88faa1bc424bad7ae: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:45.663662 5022 consensus_queue.cc:1035] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:47.924844 4041 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260929 14:10:47.925622 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:47.932833 4694 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4313956feee54744b19e2bf91abfa6c6"
dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
mode: GRACEFUL
new_leader_uuid: "376e574c0386490bbc10b1c68119b938"
from {username='slave'} at 127.0.0.1:34782
I20260929 14:10:47.933485 4694 raft_consensus.cc:595] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Received request to transfer leadership to TS 376e574c0386490bbc10b1c68119b938
I20260929 14:10:47.935660 4694 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "57edb5a9c58445679fb36c0d352112e3"
dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
mode: GRACEFUL
new_leader_uuid: "dab47308abbe40bcb7bb9eb70781192a"
from {username='slave'} at 127.0.0.1:34782
I20260929 14:10:47.936096 4694 raft_consensus.cc:595] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 3 LEADER]: Received request to transfer leadership to TS dab47308abbe40bcb7bb9eb70781192a
I20260929 14:10:47.938102 4694 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "97517573ba144283b9b7ec110600bba5"
dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
mode: GRACEFUL
new_leader_uuid: "376e574c0386490bbc10b1c68119b938"
from {username='slave'} at 127.0.0.1:34782
I20260929 14:10:47.938496 4694 raft_consensus.cc:595] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 1 LEADER]: Received request to transfer leadership to TS 376e574c0386490bbc10b1c68119b938
I20260929 14:10:47.940495 4694 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f320a6a2097b4bf2927559b5797087ca"
dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
mode: GRACEFUL
new_leader_uuid: "dab47308abbe40bcb7bb9eb70781192a"
from {username='slave'} at 127.0.0.1:34782
I20260929 14:10:47.940892 4694 raft_consensus.cc:595] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Received request to transfer leadership to TS dab47308abbe40bcb7bb9eb70781192a
I20260929 14:10:47.941670 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260929 14:10:47.942082 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:48.168360 5008 raft_consensus.cc:979] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae: : Instructing follower 376e574c0386490bbc10b1c68119b938 to start an election
I20260929 14:10:48.168814 5036 raft_consensus.cc:1067] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 1 LEADER]: Signalling peer 376e574c0386490bbc10b1c68119b938 to start an election
I20260929 14:10:48.170194 4769 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "97517573ba144283b9b7ec110600bba5"
dest_uuid: "376e574c0386490bbc10b1c68119b938"
from {username='slave'} at 127.0.0.1:37864
I20260929 14:10:48.170701 4769 raft_consensus.cc:482] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:48.171001 4769 raft_consensus.cc:3037] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:48.173741 5008 raft_consensus.cc:979] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae: : Instructing follower 376e574c0386490bbc10b1c68119b938 to start an election
I20260929 14:10:48.174186 5008 raft_consensus.cc:1067] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Signalling peer 376e574c0386490bbc10b1c68119b938 to start an election
I20260929 14:10:48.175491 4770 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4313956feee54744b19e2bf91abfa6c6"
dest_uuid: "376e574c0386490bbc10b1c68119b938"
from {username='slave'} at 127.0.0.1:37864
I20260929 14:10:48.175966 4770 raft_consensus.cc:482] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:48.176263 4770 raft_consensus.cc:3037] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:48.176378 4769 raft_consensus.cc:504] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:48.178196 4769 leader_election.cc:288] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 2 election: Requested vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:48.179175 4694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97517573ba144283b9b7ec110600bba5" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
I20260929 14:10:48.179186 4844 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97517573ba144283b9b7ec110600bba5" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:48.179781 4694 raft_consensus.cc:3032] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:10:48.179865 4844 raft_consensus.cc:3037] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:48.180071 4694 raft_consensus.cc:729] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:48.180912 4694 consensus_queue.cc:260] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:48.181960 4694 raft_consensus.cc:3037] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:10:48.182646 4770 raft_consensus.cc:504] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:48.184777 4770 leader_election.cc:288] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 3 election: Requested vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), dab47308abbe40bcb7bb9eb70781192a (127.3.242.67:40873)
I20260929 14:10:48.185351 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4313956feee54744b19e2bf91abfa6c6" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
I20260929 14:10:48.185993 4695 raft_consensus.cc:3032] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:10:48.185956 4842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4313956feee54744b19e2bf91abfa6c6" candidate_uuid: "376e574c0386490bbc10b1c68119b938" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
I20260929 14:10:48.186246 4844 raft_consensus.cc:2454] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 2.
I20260929 14:10:48.186463 4695 raft_consensus.cc:729] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:48.186612 4842 raft_consensus.cc:3037] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:48.187247 4695 consensus_queue.cc:260] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:48.188258 4695 raft_consensus.cc:3037] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:48.188380 4694 raft_consensus.cc:2454] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 2.
I20260929 14:10:48.189647 4733 leader_election.cc:302] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:48.190490 5009 raft_consensus.cc:2781] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:10:48.192916 4842 raft_consensus.cc:2454] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 3.
I20260929 14:10:48.192911 5009 raft_consensus.cc:686] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [term 2 LEADER]: Becoming Leader. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Running, Role: LEADER
I20260929 14:10:48.194140 4733 leader_election.cc:302] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:48.194146 5009 consensus_queue.cc:237] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:48.195170 5037 raft_consensus.cc:2781] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:10:48.195868 4695 raft_consensus.cc:2454] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 376e574c0386490bbc10b1c68119b938 in term 3.
I20260929 14:10:48.197280 5037 raft_consensus.cc:686] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 3 LEADER]: Becoming Leader. State: Replica: 376e574c0386490bbc10b1c68119b938, State: Running, Role: LEADER
I20260929 14:10:48.198128 5037 consensus_queue.cc:237] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:48.202916 4593 catalog_manager.cc:5495] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 reported cstate change: term changed from 1 to 2, leader changed from 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65) to 376e574c0386490bbc10b1c68119b938 (127.3.242.66). New cstate: current_term: 2 leader_uuid: "376e574c0386490bbc10b1c68119b938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:48.217507 4593 catalog_manager.cc:5495] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 reported cstate change: term changed from 2 to 3, leader changed from 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65) to 376e574c0386490bbc10b1c68119b938 (127.3.242.66). New cstate: current_term: 3 leader_uuid: "376e574c0386490bbc10b1c68119b938" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: UNKNOWN } } }
I20260929 14:10:48.294445 5036 raft_consensus.cc:979] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae: : Instructing follower dab47308abbe40bcb7bb9eb70781192a to start an election
I20260929 14:10:48.294871 5036 raft_consensus.cc:1067] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Signalling peer dab47308abbe40bcb7bb9eb70781192a to start an election
I20260929 14:10:48.296066 4842 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f320a6a2097b4bf2927559b5797087ca"
dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
from {username='slave'} at 127.0.0.1:54058
I20260929 14:10:48.296437 4842 raft_consensus.cc:482] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:48.296639 4842 raft_consensus.cc:3037] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:48.300395 4842 raft_consensus.cc:504] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:48.301957 4842 leader_election.cc:288] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 3 election: Requested vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601)
I20260929 14:10:48.302863 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f320a6a2097b4bf2927559b5797087ca" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
I20260929 14:10:48.302948 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f320a6a2097b4bf2927559b5797087ca" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:48.303544 4695 raft_consensus.cc:3032] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:10:48.303579 4770 raft_consensus.cc:3037] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:48.303992 4695 raft_consensus.cc:729] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:48.304746 4695 consensus_queue.cc:260] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:48.305948 4695 raft_consensus.cc:3037] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:10:48.310174 4770 raft_consensus.cc:2454] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 3.
I20260929 14:10:48.311389 4807 leader_election.cc:302] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:48.311923 4695 raft_consensus.cc:2454] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 3.
I20260929 14:10:48.312103 5007 raft_consensus.cc:2781] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:10:48.313453 5007 raft_consensus.cc:686] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [term 3 LEADER]: Becoming Leader. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Running, Role: LEADER
I20260929 14:10:48.314239 5007 consensus_queue.cc:237] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:48.320762 4593 catalog_manager.cc:5495] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a reported cstate change: term changed from 2 to 3, leader changed from 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65) to dab47308abbe40bcb7bb9eb70781192a (127.3.242.67). New cstate: current_term: 3 leader_uuid: "dab47308abbe40bcb7bb9eb70781192a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: HEALTHY } } }
I20260929 14:10:48.482055 5008 raft_consensus.cc:979] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae: : Instructing follower dab47308abbe40bcb7bb9eb70781192a to start an election
I20260929 14:10:48.482463 5036 raft_consensus.cc:1067] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 3 LEADER]: Signalling peer dab47308abbe40bcb7bb9eb70781192a to start an election
I20260929 14:10:48.483537 4842 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "57edb5a9c58445679fb36c0d352112e3"
dest_uuid: "dab47308abbe40bcb7bb9eb70781192a"
from {username='slave'} at 127.0.0.1:54058
I20260929 14:10:48.483966 4842 raft_consensus.cc:482] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:10:48.484210 4842 raft_consensus.cc:3037] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:48.488049 4842 raft_consensus.cc:504] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:48.489452 4842 leader_election.cc:288] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 4 election: Requested vote from peers 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523), 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601)
I20260929 14:10:48.490412 4695 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57edb5a9c58445679fb36c0d352112e3" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5898905d9cda47a88faa1bc424bad7ae"
I20260929 14:10:48.490621 4770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57edb5a9c58445679fb36c0d352112e3" candidate_uuid: "dab47308abbe40bcb7bb9eb70781192a" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "376e574c0386490bbc10b1c68119b938"
I20260929 14:10:48.491010 4695 raft_consensus.cc:3032] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:10:48.491137 4770 raft_consensus.cc:3037] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:48.491353 4695 raft_consensus.cc:729] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5898905d9cda47a88faa1bc424bad7ae, State: Running, Role: LEADER
I20260929 14:10:48.492103 4695 consensus_queue.cc:260] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:48.493295 4695 raft_consensus.cc:3037] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:10:48.497596 4770 raft_consensus.cc:2454] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 4.
I20260929 14:10:48.498646 4807 leader_election.cc:302] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 376e574c0386490bbc10b1c68119b938, dab47308abbe40bcb7bb9eb70781192a; no voters:
I20260929 14:10:48.499150 4695 raft_consensus.cc:2454] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab47308abbe40bcb7bb9eb70781192a in term 4.
I20260929 14:10:48.499291 5007 raft_consensus.cc:2781] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:10:48.499758 5007 raft_consensus.cc:686] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 4 LEADER]: Becoming Leader. State: Replica: dab47308abbe40bcb7bb9eb70781192a, State: Running, Role: LEADER
I20260929 14:10:48.500425 5007 consensus_queue.cc:237] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } }
I20260929 14:10:48.506735 4593 catalog_manager.cc:5495] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a reported cstate change: term changed from 3 to 4, leader changed from 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65) to dab47308abbe40bcb7bb9eb70781192a (127.3.242.67). New cstate: current_term: 4 leader_uuid: "dab47308abbe40bcb7bb9eb70781192a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 } health_report { overall_health: HEALTHY } } }
I20260929 14:10:48.613736 4842 raft_consensus.cc:1261] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Refusing update from remote peer 376e574c0386490bbc10b1c68119b938: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:48.614825 5037 consensus_queue.cc:1035] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:48.624501 4695 raft_consensus.cc:1261] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Refusing update from remote peer 376e574c0386490bbc10b1c68119b938: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:48.626716 5037 consensus_queue.cc:1035] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:48.645840 4695 raft_consensus.cc:1261] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Refusing update from remote peer 376e574c0386490bbc10b1c68119b938: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:48.646966 5037 consensus_queue.cc:1035] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:48.655393 4842 raft_consensus.cc:1261] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Refusing update from remote peer 376e574c0386490bbc10b1c68119b938: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:10:48.656821 5037 consensus_queue.cc:1035] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab47308abbe40bcb7bb9eb70781192a" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:10:48.844185 4695 raft_consensus.cc:1261] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Refusing update from remote peer dab47308abbe40bcb7bb9eb70781192a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:48.845196 5007 consensus_queue.cc:1035] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:48.851451 4770 raft_consensus.cc:1261] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Refusing update from remote peer dab47308abbe40bcb7bb9eb70781192a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:10:48.853158 5007 consensus_queue.cc:1035] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:10:48.943004 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:48.951017 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:48.951378 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:48.985680 4695 raft_consensus.cc:1261] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 4 FOLLOWER]: Refusing update from remote peer dab47308abbe40bcb7bb9eb70781192a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260929 14:10:48.987126 5007 consensus_queue.cc:1035] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5898905d9cda47a88faa1bc424bad7ae" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 35523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:10:48.998207 4770 raft_consensus.cc:1261] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 4 FOLLOWER]: Refusing update from remote peer dab47308abbe40bcb7bb9eb70781192a: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260929 14:10:48.999919 5007 consensus_queue.cc:1035] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [LEADER]: Connected to new peer: Peer: permanent_uuid: "376e574c0386490bbc10b1c68119b938" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 42601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:10:49.952342 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:49.958480 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:49.958895 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:50.959754 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:50.965294 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:50.965684 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:51.966420 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:51.971361 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:51.971725 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:52.972434 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:52.977451 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:52.977826 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:53.978511 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:53.983973 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:53.984347 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:54.985075 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:54.990010 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:54.990404 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:55.991139 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:55.996034 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:55.996461 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:56.997169 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:10:57.002085 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260929 14:10:57.002408 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:10:58.034715 4041 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260929 14:10:58.035192 4041 tablet_server.cc:178] TabletServer@127.3.242.65:0 shutting down...
W20260929 14:10:58.050395 4808 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 4 similar messages]
W20260929 14:10:58.056705 4808 consensus_peers.cc:487] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a -> Peer 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523): Couldn't send request to peer 5898905d9cda47a88faa1bc424bad7ae. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:10:58.072803 4041 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260929 14:10:58.073526 4041 tablet_replica.cc:330] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae: stopping tablet replica
I20260929 14:10:58.074143 4041 raft_consensus.cc:2229] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 4 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.074592 4041 raft_consensus.cc:2258] T 57edb5a9c58445679fb36c0d352112e3 P 5898905d9cda47a88faa1bc424bad7ae [term 4 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.076488 4041 tablet_replica.cc:330] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae: stopping tablet replica
I20260929 14:10:58.077069 4041 raft_consensus.cc:2229] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.077771 4041 raft_consensus.cc:2258] T 4313956feee54744b19e2bf91abfa6c6 P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.080255 4041 tablet_replica.cc:330] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae: stopping tablet replica
W20260929 14:10:58.080349 4734 consensus_peers.cc:487] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 -> Peer 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523): Couldn't send request to peer 5898905d9cda47a88faa1bc424bad7ae. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:10:58.080935 4041 raft_consensus.cc:2229] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.081385 4041 raft_consensus.cc:2258] T 7d06d6f2848041fcbbe5f57ce706e118 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.083105 4041 tablet_replica.cc:330] T 6ae676f8e6814c8296d465f3764c77e9 P 5898905d9cda47a88faa1bc424bad7ae: stopping tablet replica
I20260929 14:10:58.083590 4041 raft_consensus.cc:2229] T 6ae676f8e6814c8296d465f3764c77e9 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.083949 4041 raft_consensus.cc:2258] T 6ae676f8e6814c8296d465f3764c77e9 P 5898905d9cda47a88faa1bc424bad7ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.085564 4041 tablet_replica.cc:330] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae: stopping tablet replica
I20260929 14:10:58.086062 4041 raft_consensus.cc:2229] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Raft consensus shutting down.
I20260929 14:10:58.086794 4041 raft_consensus.cc:2258] T b10fa99b33bb43399376921aab640714 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.088246 4041 tablet_replica.cc:330] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae: stopping tablet replica
I20260929 14:10:58.088721 4041 raft_consensus.cc:2229] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 2 LEADER]: Raft consensus shutting down.
I20260929 14:10:58.089663 4041 raft_consensus.cc:2258] T 61659e7e13f94f06ade1d6a63d40c865 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.091356 4041 tablet_replica.cc:330] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae: stopping tablet replica
I20260929 14:10:58.091857 4041 raft_consensus.cc:2229] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 3 LEADER]: Raft consensus shutting down.
I20260929 14:10:58.092549 4041 raft_consensus.cc:2258] T 0474cce108714cf3a6811d25f5ce09f4 P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.094147 4041 tablet_replica.cc:330] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae: stopping tablet replica
I20260929 14:10:58.094612 4041 raft_consensus.cc:2229] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 4 LEADER]: Raft consensus shutting down.
I20260929 14:10:58.095317 4041 raft_consensus.cc:2258] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 5898905d9cda47a88faa1bc424bad7ae [term 4 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.096858 4041 tablet_replica.cc:330] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae: stopping tablet replica
I20260929 14:10:58.097373 4041 raft_consensus.cc:2229] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.097757 4041 raft_consensus.cc:2258] T f320a6a2097b4bf2927559b5797087ca P 5898905d9cda47a88faa1bc424bad7ae [term 3 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.099210 4041 tablet_replica.cc:330] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae: stopping tablet replica
I20260929 14:10:58.099648 4041 raft_consensus.cc:2229] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.100080 4041 raft_consensus.cc:2258] T 97517573ba144283b9b7ec110600bba5 P 5898905d9cda47a88faa1bc424bad7ae [term 2 FOLLOWER]: Raft consensus is shut down!
W20260929 14:10:58.129403 4734 consensus_peers.cc:487] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 -> Peer 5898905d9cda47a88faa1bc424bad7ae (127.3.242.65:35523): Couldn't send request to peer 5898905d9cda47a88faa1bc424bad7ae. Status: Network error: Client connection negotiation failed: client connection to 127.3.242.65:35523: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 14:10:58.135809 4041 tablet_server.cc:194] TabletServer@127.3.242.65:0 shutdown complete.
I20260929 14:10:58.153402 4041 tablet_server.cc:178] TabletServer@127.3.242.66:0 shutting down...
I20260929 14:10:58.178980 4041 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260929 14:10:58.179593 4041 tablet_replica.cc:330] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938: stopping tablet replica
I20260929 14:10:58.180166 4041 raft_consensus.cc:2229] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [term 2 LEADER]: Raft consensus shutting down.
I20260929 14:10:58.180902 4041 raft_consensus.cc:2258] T 97517573ba144283b9b7ec110600bba5 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.182617 4041 tablet_replica.cc:330] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938: stopping tablet replica
I20260929 14:10:58.183120 4041 raft_consensus.cc:2229] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.183534 4041 raft_consensus.cc:2258] T f320a6a2097b4bf2927559b5797087ca P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.185047 4041 tablet_replica.cc:330] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938: stopping tablet replica
I20260929 14:10:58.185581 4041 raft_consensus.cc:2229] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.185976 4041 raft_consensus.cc:2258] T 7d06d6f2848041fcbbe5f57ce706e118 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.187794 4041 tablet_replica.cc:330] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938: stopping tablet replica
I20260929 14:10:58.188287 4041 raft_consensus.cc:2229] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 3 LEADER]: Raft consensus shutting down.
I20260929 14:10:58.189010 4041 raft_consensus.cc:2258] T 4313956feee54744b19e2bf91abfa6c6 P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.190652 4041 tablet_replica.cc:330] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938: stopping tablet replica
I20260929 14:10:58.191125 4041 raft_consensus.cc:2229] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:10:58.191859 4041 raft_consensus.cc:2258] T 6ae676f8e6814c8296d465f3764c77e9 P 376e574c0386490bbc10b1c68119b938 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.193629 4041 tablet_replica.cc:330] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938: stopping tablet replica
I20260929 14:10:58.194095 4041 raft_consensus.cc:2229] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.194509 4041 raft_consensus.cc:2258] T b10fa99b33bb43399376921aab640714 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.196049 4041 tablet_replica.cc:330] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938: stopping tablet replica
I20260929 14:10:58.196507 4041 raft_consensus.cc:2229] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.196887 4041 raft_consensus.cc:2258] T 57edb5a9c58445679fb36c0d352112e3 P 376e574c0386490bbc10b1c68119b938 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.198575 4041 tablet_replica.cc:330] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938: stopping tablet replica
I20260929 14:10:58.199002 4041 raft_consensus.cc:2229] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.199380 4041 raft_consensus.cc:2258] T 61659e7e13f94f06ade1d6a63d40c865 P 376e574c0386490bbc10b1c68119b938 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.200872 4041 tablet_replica.cc:330] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938: stopping tablet replica
I20260929 14:10:58.201381 4041 raft_consensus.cc:2229] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.201825 4041 raft_consensus.cc:2258] T 0474cce108714cf3a6811d25f5ce09f4 P 376e574c0386490bbc10b1c68119b938 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.203279 4041 tablet_replica.cc:330] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938: stopping tablet replica
I20260929 14:10:58.203686 4041 raft_consensus.cc:2229] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.204068 4041 raft_consensus.cc:2258] T 4d6ebbf98eac48d392118aa29cdd0cb8 P 376e574c0386490bbc10b1c68119b938 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.232636 4041 tablet_server.cc:194] TabletServer@127.3.242.66:0 shutdown complete.
I20260929 14:10:58.251142 4041 tablet_server.cc:178] TabletServer@127.3.242.67:0 shutting down...
I20260929 14:10:58.275422 4041 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260929 14:10:58.276033 4041 tablet_replica.cc:330] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a: stopping tablet replica
I20260929 14:10:58.276690 4041 raft_consensus.cc:2229] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 4 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.277172 4041 raft_consensus.cc:2258] T 4d6ebbf98eac48d392118aa29cdd0cb8 P dab47308abbe40bcb7bb9eb70781192a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.278934 4041 tablet_replica.cc:330] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a: stopping tablet replica
I20260929 14:10:58.279507 4041 raft_consensus.cc:2229] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.279925 4041 raft_consensus.cc:2258] T 61659e7e13f94f06ade1d6a63d40c865 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.282423 4041 tablet_replica.cc:330] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a: stopping tablet replica
I20260929 14:10:58.283159 4041 raft_consensus.cc:2229] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.283684 4041 raft_consensus.cc:2258] T 0474cce108714cf3a6811d25f5ce09f4 P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Raft consensus is shut down!
W20260929 14:10:58.284260 4807 consensus_peers.cc:487] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a -> Peer 376e574c0386490bbc10b1c68119b938 (127.3.242.66:42601): Couldn't send request to peer 376e574c0386490bbc10b1c68119b938. Status: Network error: Client connection negotiation failed: client connection to 127.3.242.66:42601: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 14:10:58.285676 4041 tablet_replica.cc:330] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a: stopping tablet replica
I20260929 14:10:58.286114 4041 raft_consensus.cc:2229] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.286471 4041 raft_consensus.cc:2258] T b10fa99b33bb43399376921aab640714 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.287997 4041 tablet_replica.cc:330] T 6ae676f8e6814c8296d465f3764c77e9 P dab47308abbe40bcb7bb9eb70781192a: stopping tablet replica
I20260929 14:10:58.288477 4041 raft_consensus.cc:2229] T 6ae676f8e6814c8296d465f3764c77e9 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.288842 4041 raft_consensus.cc:2258] T 6ae676f8e6814c8296d465f3764c77e9 P dab47308abbe40bcb7bb9eb70781192a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.290517 4041 tablet_replica.cc:330] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a: stopping tablet replica
I20260929 14:10:58.290933 4041 raft_consensus.cc:2229] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.291294 4041 raft_consensus.cc:2258] T 4313956feee54744b19e2bf91abfa6c6 P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.293200 4041 tablet_replica.cc:330] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a: stopping tablet replica
I20260929 14:10:58.293732 4041 raft_consensus.cc:2229] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [term 2 LEADER]: Raft consensus shutting down.
I20260929 14:10:58.294483 4041 raft_consensus.cc:2258] T 7d06d6f2848041fcbbe5f57ce706e118 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.296401 4041 tablet_replica.cc:330] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a: stopping tablet replica
I20260929 14:10:58.296832 4041 raft_consensus.cc:2229] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [term 3 LEADER]: Raft consensus shutting down.
I20260929 14:10:58.297504 4041 raft_consensus.cc:2258] T f320a6a2097b4bf2927559b5797087ca P dab47308abbe40bcb7bb9eb70781192a [term 3 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.299021 4041 tablet_replica.cc:330] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a: stopping tablet replica
I20260929 14:10:58.299434 4041 raft_consensus.cc:2229] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260929 14:10:58.299795 4041 raft_consensus.cc:2258] T 97517573ba144283b9b7ec110600bba5 P dab47308abbe40bcb7bb9eb70781192a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.301306 4041 tablet_replica.cc:330] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a: stopping tablet replica
I20260929 14:10:58.301700 4041 raft_consensus.cc:2229] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 4 LEADER]: Raft consensus shutting down.
I20260929 14:10:58.302309 4041 raft_consensus.cc:2258] T 57edb5a9c58445679fb36c0d352112e3 P dab47308abbe40bcb7bb9eb70781192a [term 4 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.330581 4041 tablet_server.cc:194] TabletServer@127.3.242.67:0 shutdown complete.
I20260929 14:10:58.349509 4041 master.cc:537] Master@127.3.242.126:40359 shutting down...
I20260929 14:10:58.374598 4041 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [term 1 LEADER]: Raft consensus shutting down.
I20260929 14:10:58.375128 4041 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:10:58.375403 4041 tablet_replica.cc:330] T 00000000000000000000000000000000 P 225b6703403a47c8a9c887c7f451b0a2: stopping tablet replica
I20260929 14:10:58.394016 4041 master.cc:552] Master@127.3.242.126:40359 shutdown complete.
376e574c0386490bbc10b1c68119b938 4
5898905d9cda47a88faa1bc424bad7ae 5
dab47308abbe40bcb7bb9eb70781192a 1
376e574c0386490bbc10b1c68119b938 4
5898905d9cda47a88faa1bc424bad7ae 3
dab47308abbe40bcb7bb9eb70781192a 3
376e574c0386490bbc10b1c68119b938 1
5898905d9cda47a88faa1bc424bad7ae 8
dab47308abbe40bcb7bb9eb70781192a 1
376e574c0386490bbc10b1c68119b938 3
5898905d9cda47a88faa1bc424bad7ae 4
dab47308abbe40bcb7bb9eb70781192a 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28735 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260929 14:10:58.436894 4041 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.3.242.126:41327
I20260929 14:10:58.438019 4041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:10:58.444973 5077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:10:58.445456 4041 server_base.cc:1034] running on GCE node
W20260929 14:10:58.445006 5075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:58.444996 5074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:10:58.447045 4041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 14:10:58.447263 4041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 14:10:58.447430 4041 hybrid_clock.cc:648] HybridClock initialized: now 1790691058447415 us; error 0 us; skew 500 ppm
I20260929 14:10:58.447955 4041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 14:10:58.450949 4041 webserver.cc:460] Webserver started at http://127.3.242.126:38433/ using document root <none> and password file <none>
I20260929 14:10:58.451485 4041 fs_manager.cc:359] Metadata directory not provided
I20260929 14:10:58.451738 4041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:10:58.452014 4041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:10:58.453181 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/master-0-root/instance:
uuid: "101c95b7e4d04f699d0fa198361f4d29"
format_stamp: "Formatted at 2026-09-29 14:10:58 on dist-test-slave-6gv9"
I20260929 14:10:58.458256 4041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:10:58.461630 5082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:58.462509 4041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20260929 14:10:58.462740 4041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/master-0-root
uuid: "101c95b7e4d04f699d0fa198361f4d29"
format_stamp: "Formatted at 2026-09-29 14:10:58 on dist-test-slave-6gv9"
I20260929 14:10:58.462980 4041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:10:58.479153 4041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:10:58.480638 4041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:10:58.518711 4041 rpc_server.cc:307] RPC server started. Bound to: 127.3.242.126:41327
I20260929 14:10:58.518831 5133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.242.126:41327 every 8 connection(s)
I20260929 14:10:58.522611 5134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:58.532002 5134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29: Bootstrap starting.
I20260929 14:10:58.536120 5134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:58.540167 5134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29: No bootstrap required, opened a new log
I20260929 14:10:58.542313 5134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101c95b7e4d04f699d0fa198361f4d29" member_type: VOTER }
I20260929 14:10:58.542707 5134 raft_consensus.cc:374] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:58.542964 5134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 101c95b7e4d04f699d0fa198361f4d29, State: Initialized, Role: FOLLOWER
I20260929 14:10:58.543596 5134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101c95b7e4d04f699d0fa198361f4d29" member_type: VOTER }
I20260929 14:10:58.543982 5134 raft_consensus.cc:388] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 14:10:58.544250 5134 raft_consensus.cc:482] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 14:10:58.544546 5134 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:10:58.548590 5134 raft_consensus.cc:504] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101c95b7e4d04f699d0fa198361f4d29" member_type: VOTER }
I20260929 14:10:58.549109 5134 leader_election.cc:302] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 101c95b7e4d04f699d0fa198361f4d29; no voters:
I20260929 14:10:58.550257 5134 leader_election.cc:288] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 14:10:58.550558 5137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:10:58.551904 5137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [term 1 LEADER]: Becoming Leader. State: Replica: 101c95b7e4d04f699d0fa198361f4d29, State: Running, Role: LEADER
I20260929 14:10:58.552492 5137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101c95b7e4d04f699d0fa198361f4d29" member_type: VOTER }
I20260929 14:10:58.553191 5134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 14:10:58.557533 5138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "101c95b7e4d04f699d0fa198361f4d29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101c95b7e4d04f699d0fa198361f4d29" member_type: VOTER } }
I20260929 14:10:58.557796 5139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 101c95b7e4d04f699d0fa198361f4d29. Latest consensus state: current_term: 1 leader_uuid: "101c95b7e4d04f699d0fa198361f4d29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "101c95b7e4d04f699d0fa198361f4d29" member_type: VOTER } }
I20260929 14:10:58.558348 5138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [sys.catalog]: This master's current role is: LEADER
I20260929 14:10:58.558509 5139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29 [sys.catalog]: This master's current role is: LEADER
I20260929 14:10:58.561481 5145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 14:10:58.566401 5145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 14:10:58.569283 4041 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260929 14:10:58.574987 5145 catalog_manager.cc:1347] Generated new cluster ID: 2a9160c8aa544704a461589798066b71
I20260929 14:10:58.575284 5145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 14:10:58.611481 5145 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 14:10:58.613129 5145 catalog_manager.cc:1504] Loading token signing keys...
I20260929 14:10:58.627641 5145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29: Generated new TSK 0
I20260929 14:10:58.628307 5145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 14:10:58.636224 4041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:10:58.642244 5155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:58.643306 5156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:10:58.645813 4041 server_base.cc:1034] running on GCE node
W20260929 14:10:58.646092 5158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:10:58.646956 4041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 14:10:58.647184 4041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 14:10:58.647341 4041 hybrid_clock.cc:648] HybridClock initialized: now 1790691058647326 us; error 0 us; skew 500 ppm
I20260929 14:10:58.647864 4041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 14:10:58.650982 4041 webserver.cc:460] Webserver started at http://127.3.242.65:36501/ using document root <none> and password file <none>
I20260929 14:10:58.651507 4041 fs_manager.cc:359] Metadata directory not provided
I20260929 14:10:58.651717 4041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:10:58.651976 4041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:10:58.653043 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-0-root/instance:
uuid: "ac17d3b553b74c9da39e0ab307addacd"
format_stamp: "Formatted at 2026-09-29 14:10:58 on dist-test-slave-6gv9"
I20260929 14:10:58.657567 4041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 14:10:58.660744 5163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:58.661619 4041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 14:10:58.661908 4041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-0-root
uuid: "ac17d3b553b74c9da39e0ab307addacd"
format_stamp: "Formatted at 2026-09-29 14:10:58 on dist-test-slave-6gv9"
I20260929 14:10:58.662236 4041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:10:58.676898 4041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:10:58.678119 4041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:10:58.679565 4041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:10:58.681829 4041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:10:58.682061 4041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:58.682323 4041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:10:58.682502 4041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:58.723254 4041 rpc_server.cc:307] RPC server started. Bound to: 127.3.242.65:45147
I20260929 14:10:58.723351 5225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.242.65:45147 every 8 connection(s)
I20260929 14:10:58.728068 4041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:10:58.735977 5231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:58.737818 5230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:58.739373 5233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:10:58.740959 4041 server_base.cc:1034] running on GCE node
I20260929 14:10:58.742141 4041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 14:10:58.742321 4041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 14:10:58.742530 4041 hybrid_clock.cc:648] HybridClock initialized: now 1790691058742515 us; error 0 us; skew 500 ppm
I20260929 14:10:58.743040 4041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 14:10:58.745163 5226 heartbeater.cc:347] Connected to a master server at 127.3.242.126:41327
I20260929 14:10:58.745607 5226 heartbeater.cc:464] Registering TS with master...
I20260929 14:10:58.745779 4041 webserver.cc:460] Webserver started at http://127.3.242.66:44539/ using document root <none> and password file <none>
I20260929 14:10:58.746316 4041 fs_manager.cc:359] Metadata directory not provided
I20260929 14:10:58.746372 5226 heartbeater.cc:511] Master 127.3.242.126:41327 requested a full tablet report, sending...
I20260929 14:10:58.746526 4041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:10:58.746853 4041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:10:58.747942 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-1-root/instance:
uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
format_stamp: "Formatted at 2026-09-29 14:10:58 on dist-test-slave-6gv9"
I20260929 14:10:58.748427 5099 ts_manager.cc:194] Registered new tserver with Master: ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147)
I20260929 14:10:58.750428 5099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52726
I20260929 14:10:58.752683 4041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 14:10:58.756033 5238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:58.756891 4041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 14:10:58.757294 4041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-1-root
uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
format_stamp: "Formatted at 2026-09-29 14:10:58 on dist-test-slave-6gv9"
I20260929 14:10:58.757712 4041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:10:58.781172 4041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:10:58.782369 4041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:10:58.783772 4041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:10:58.785902 4041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:10:58.786110 4041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:58.786348 4041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:10:58.786517 4041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:58.826478 4041 rpc_server.cc:307] RPC server started. Bound to: 127.3.242.66:43473
I20260929 14:10:58.826604 5300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.242.66:43473 every 8 connection(s)
I20260929 14:10:58.830976 4041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:10:58.838187 5304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:58.841482 5307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:10:58.840787 5305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:10:58.844182 4041 server_base.cc:1034] running on GCE node
I20260929 14:10:58.845114 4041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260929 14:10:58.845264 5301 heartbeater.cc:347] Connected to a master server at 127.3.242.126:41327
W20260929 14:10:58.845422 4041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 14:10:58.845636 5301 heartbeater.cc:464] Registering TS with master...
I20260929 14:10:58.845633 4041 hybrid_clock.cc:648] HybridClock initialized: now 1790691058845615 us; error 0 us; skew 500 ppm
I20260929 14:10:58.846477 4041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 14:10:58.846560 5301 heartbeater.cc:511] Master 127.3.242.126:41327 requested a full tablet report, sending...
I20260929 14:10:58.848744 5099 ts_manager.cc:194] Registered new tserver with Master: c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:10:58.849691 4041 webserver.cc:460] Webserver started at http://127.3.242.67:42851/ using document root <none> and password file <none>
I20260929 14:10:58.850288 4041 fs_manager.cc:359] Metadata directory not provided
I20260929 14:10:58.850526 4041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:10:58.850533 5099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52736
I20260929 14:10:58.851056 4041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:10:58.852507 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-2-root/instance:
uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
format_stamp: "Formatted at 2026-09-29 14:10:58 on dist-test-slave-6gv9"
I20260929 14:10:58.856951 4041 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260929 14:10:58.860282 5312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:58.861011 4041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 14:10:58.861310 4041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-2-root
uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
format_stamp: "Formatted at 2026-09-29 14:10:58 on dist-test-slave-6gv9"
I20260929 14:10:58.861591 4041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:10:58.878311 4041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:10:58.879410 4041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:10:58.880787 4041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:10:58.882911 4041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:10:58.883122 4041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:58.883369 4041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:10:58.883535 4041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:10:58.962961 4041 rpc_server.cc:307] RPC server started. Bound to: 127.3.242.67:42639
I20260929 14:10:58.963063 5375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.242.67:42639 every 8 connection(s)
I20260929 14:10:58.982574 5376 heartbeater.cc:347] Connected to a master server at 127.3.242.126:41327
I20260929 14:10:58.982921 5376 heartbeater.cc:464] Registering TS with master...
I20260929 14:10:58.983744 5376 heartbeater.cc:511] Master 127.3.242.126:41327 requested a full tablet report, sending...
I20260929 14:10:58.986948 5099 ts_manager.cc:194] Registered new tserver with Master: d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:10:58.987114 4041 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.018930105s
I20260929 14:10:58.989203 5099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:52750
I20260929 14:10:58.990404 4041 test_util.cc:268] Using random seed: -1065796306
I20260929 14:10:59.017692 5099 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52762:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 14:10:59.024369 5099 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 14:10:59.252298 5186 tablet_service.cc:1467] Processing CreateTablet for tablet 599240a72b454fca841864f021dbbc64 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260929 14:10:59.252797 5191 tablet_service.cc:1467] Processing CreateTablet for tablet 88e8219d06734c04acdce83c7b9c1c53 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260929 14:10:59.254237 5186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599240a72b454fca841864f021dbbc64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.255319 5190 tablet_service.cc:1467] Processing CreateTablet for tablet b7924b2689dc46c98af8c4344a1e1e33 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260929 14:10:59.255611 5266 tablet_service.cc:1467] Processing CreateTablet for tablet 88e8219d06734c04acdce83c7b9c1c53 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260929 14:10:59.256608 5190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7924b2689dc46c98af8c4344a1e1e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.258426 5189 tablet_service.cc:1467] Processing CreateTablet for tablet 4b277670ba5344b493c046b2a0864597 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260929 14:10:59.258541 5265 tablet_service.cc:1467] Processing CreateTablet for tablet b7924b2689dc46c98af8c4344a1e1e33 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260929 14:10:59.259696 5189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b277670ba5344b493c046b2a0864597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.260905 5264 tablet_service.cc:1467] Processing CreateTablet for tablet 4b277670ba5344b493c046b2a0864597 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260929 14:10:59.262423 5188 tablet_service.cc:1467] Processing CreateTablet for tablet 05a81ea24145409b82fe949d849ab73d (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260929 14:10:59.263252 5263 tablet_service.cc:1467] Processing CreateTablet for tablet 05a81ea24145409b82fe949d849ab73d (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260929 14:10:59.263736 5188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a81ea24145409b82fe949d849ab73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.265617 5191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e8219d06734c04acdce83c7b9c1c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.265561 5262 tablet_service.cc:1467] Processing CreateTablet for tablet 879dc5894e734d098eb2d6064e10712a (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260929 14:10:59.266507 5187 tablet_service.cc:1467] Processing CreateTablet for tablet 879dc5894e734d098eb2d6064e10712a (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260929 14:10:59.267814 5187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879dc5894e734d098eb2d6064e10712a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.267988 5261 tablet_service.cc:1467] Processing CreateTablet for tablet 599240a72b454fca841864f021dbbc64 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260929 14:10:59.270767 5260 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2589e399004a6797ef987deb148b6c (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260929 14:10:59.270838 5184 tablet_service.cc:1467] Processing CreateTablet for tablet a39f12a08e914559b1a12cbac3e3cec4 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260929 14:10:59.272406 5184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39f12a08e914559b1a12cbac3e3cec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.273836 5259 tablet_service.cc:1467] Processing CreateTablet for tablet a39f12a08e914559b1a12cbac3e3cec4 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260929 14:10:59.276111 5183 tablet_service.cc:1467] Processing CreateTablet for tablet d653defbcc7e47ca89b35b1383e4e2c8 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260929 14:10:59.277450 5183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d653defbcc7e47ca89b35b1383e4e2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.278172 5258 tablet_service.cc:1467] Processing CreateTablet for tablet d653defbcc7e47ca89b35b1383e4e2c8 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260929 14:10:59.270776 5185 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2589e399004a6797ef987deb148b6c (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260929 14:10:59.282313 5341 tablet_service.cc:1467] Processing CreateTablet for tablet 88e8219d06734c04acdce83c7b9c1c53 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260929 14:10:59.283633 5185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2589e399004a6797ef987deb148b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.283751 5341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e8219d06734c04acdce83c7b9c1c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.285507 5340 tablet_service.cc:1467] Processing CreateTablet for tablet b7924b2689dc46c98af8c4344a1e1e33 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260929 14:10:59.286788 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7924b2689dc46c98af8c4344a1e1e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.290563 5339 tablet_service.cc:1467] Processing CreateTablet for tablet 4b277670ba5344b493c046b2a0864597 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260929 14:10:59.291888 5339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b277670ba5344b493c046b2a0864597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.257115 5266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e8219d06734c04acdce83c7b9c1c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.294425 5338 tablet_service.cc:1467] Processing CreateTablet for tablet 05a81ea24145409b82fe949d849ab73d (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260929 14:10:59.295696 5338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a81ea24145409b82fe949d849ab73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.299742 5336 tablet_service.cc:1467] Processing CreateTablet for tablet 599240a72b454fca841864f021dbbc64 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260929 14:10:59.299921 5337 tablet_service.cc:1467] Processing CreateTablet for tablet 879dc5894e734d098eb2d6064e10712a (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260929 14:10:59.302103 5257 tablet_service.cc:1467] Processing CreateTablet for tablet 58eff9a0aa514df1999bdd22fb513354 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260929 14:10:59.302789 5335 tablet_service.cc:1467] Processing CreateTablet for tablet 6d2589e399004a6797ef987deb148b6c (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260929 14:10:59.306573 5182 tablet_service.cc:1467] Processing CreateTablet for tablet 58eff9a0aa514df1999bdd22fb513354 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260929 14:10:59.308053 5182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58eff9a0aa514df1999bdd22fb513354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.301151 5336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599240a72b454fca841864f021dbbc64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.322651 5333 tablet_service.cc:1467] Processing CreateTablet for tablet d653defbcc7e47ca89b35b1383e4e2c8 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260929 14:10:59.323740 5332 tablet_service.cc:1467] Processing CreateTablet for tablet 58eff9a0aa514df1999bdd22fb513354 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260929 14:10:59.324470 5333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d653defbcc7e47ca89b35b1383e4e2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.325341 5332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58eff9a0aa514df1999bdd22fb513354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.322659 5334 tablet_service.cc:1467] Processing CreateTablet for tablet a39f12a08e914559b1a12cbac3e3cec4 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260929 14:10:59.326712 5334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39f12a08e914559b1a12cbac3e3cec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.327360 5337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879dc5894e734d098eb2d6064e10712a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.329742 5335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2589e399004a6797ef987deb148b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.341101 5263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05a81ea24145409b82fe949d849ab73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.341858 5262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 879dc5894e734d098eb2d6064e10712a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.346481 5265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7924b2689dc46c98af8c4344a1e1e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.348395 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 599240a72b454fca841864f021dbbc64. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.353564 5259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a39f12a08e914559b1a12cbac3e3cec4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.354717 5257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58eff9a0aa514df1999bdd22fb513354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.358146 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b277670ba5344b493c046b2a0864597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.381664 5260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d2589e399004a6797ef987deb148b6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.382371 5258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d653defbcc7e47ca89b35b1383e4e2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.394754 5182 tablet_service.cc:1467] Processing CreateTablet for tablet 5e230699b3fb4d4d97fdb04988babce5 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260929 14:10:59.395992 5188 tablet_service.cc:1467] Processing CreateTablet for tablet 4fc85ccb35394b2d800cea4f13552756 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260929 14:10:59.410626 5188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fc85ccb35394b2d800cea4f13552756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.414091 5190 tablet_service.cc:1467] Processing CreateTablet for tablet 520249ed5be343dcb30c86d3396491a4 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260929 14:10:59.398272 5186 tablet_service.cc:1467] Processing CreateTablet for tablet e50b67ada64545a6b7525d4aba317bb0 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260929 14:10:59.415515 5186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50b67ada64545a6b7525d4aba317bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.421883 5182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e230699b3fb4d4d97fdb04988babce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.407245 5391 tablet_bootstrap.cc:492] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:10:59.409022 5189 tablet_service.cc:1467] Processing CreateTablet for tablet 678eb0db533d4bea872808b6fb39b764 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260929 14:10:59.404908 5187 tablet_service.cc:1467] Processing CreateTablet for tablet f84af62485a4405f887bb8675221641e (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260929 14:10:59.401301 5183 tablet_service.cc:1467] Processing CreateTablet for tablet dfe03c3eac5d42c280945fcb723aabc3 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260929 14:10:59.425770 5190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520249ed5be343dcb30c86d3396491a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.430768 5259 tablet_service.cc:1467] Processing CreateTablet for tablet 64181b973d25469da29a63928ede710f (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260929 14:10:59.432017 5259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64181b973d25469da29a63928ede710f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.433519 5187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f84af62485a4405f887bb8675221641e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.435765 5183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe03c3eac5d42c280945fcb723aabc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.438097 5189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 678eb0db533d4bea872808b6fb39b764. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.440212 5191 tablet_service.cc:1467] Processing CreateTablet for tablet 64181b973d25469da29a63928ede710f (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260929 14:10:59.451211 5184 tablet_service.cc:1467] Processing CreateTablet for tablet e55a9c8ae48e44c59fee506e539b3f90 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260929 14:10:59.458796 5392 tablet_bootstrap.cc:492] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:10:59.461673 5262 tablet_service.cc:1467] Processing CreateTablet for tablet 5e230699b3fb4d4d97fdb04988babce5 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260929 14:10:59.467098 5340 tablet_service.cc:1467] Processing CreateTablet for tablet 64181b973d25469da29a63928ede710f (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260929 14:10:59.467741 5266 tablet_service.cc:1467] Processing CreateTablet for tablet 4fc85ccb35394b2d800cea4f13552756 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260929 14:10:59.473644 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64181b973d25469da29a63928ede710f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.473752 5337 tablet_service.cc:1467] Processing CreateTablet for tablet 5e230699b3fb4d4d97fdb04988babce5 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260929 14:10:59.475502 5337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e230699b3fb4d4d97fdb04988babce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.477771 5191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64181b973d25469da29a63928ede710f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.478945 5333 tablet_service.cc:1467] Processing CreateTablet for tablet 4fc85ccb35394b2d800cea4f13552756 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260929 14:10:59.480157 5333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fc85ccb35394b2d800cea4f13552756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.469026 5266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fc85ccb35394b2d800cea4f13552756. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.493750 5332 tablet_service.cc:1467] Processing CreateTablet for tablet e50b67ada64545a6b7525d4aba317bb0 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260929 14:10:59.495088 5332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50b67ada64545a6b7525d4aba317bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.497830 5340 tablet_service.cc:1467] Processing CreateTablet for tablet dfe03c3eac5d42c280945fcb723aabc3 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260929 14:10:59.499157 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe03c3eac5d42c280945fcb723aabc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.500751 5333 tablet_service.cc:1467] Processing CreateTablet for tablet f84af62485a4405f887bb8675221641e (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260929 14:10:59.502115 5333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f84af62485a4405f887bb8675221641e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.505728 5182 tablet_service.cc:1467] Processing CreateTablet for tablet 465f57e1a1654ed1b40ce88c27fa52ac (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260929 14:10:59.521178 5340 tablet_service.cc:1467] Processing CreateTablet for tablet 520249ed5be343dcb30c86d3396491a4 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260929 14:10:59.522601 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520249ed5be343dcb30c86d3396491a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.522663 5264 tablet_service.cc:1467] Processing CreateTablet for tablet dfe03c3eac5d42c280945fcb723aabc3 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260929 14:10:59.524597 5333 tablet_service.cc:1467] Processing CreateTablet for tablet 678eb0db533d4bea872808b6fb39b764 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260929 14:10:59.524791 5257 tablet_service.cc:1467] Processing CreateTablet for tablet e50b67ada64545a6b7525d4aba317bb0 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260929 14:10:59.525972 5333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 678eb0db533d4bea872808b6fb39b764. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.530016 5393 tablet_bootstrap.cc:492] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:10:59.533581 5262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e230699b3fb4d4d97fdb04988babce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.542943 5340 tablet_service.cc:1467] Processing CreateTablet for tablet 465f57e1a1654ed1b40ce88c27fa52ac (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260929 14:10:59.544318 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 465f57e1a1654ed1b40ce88c27fa52ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.546948 5263 tablet_service.cc:1467] Processing CreateTablet for tablet f84af62485a4405f887bb8675221641e (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260929 14:10:59.547279 5333 tablet_service.cc:1467] Processing CreateTablet for tablet c15d9fe0c3724bfbad2156748acd468e (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260929 14:10:59.548223 5263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f84af62485a4405f887bb8675221641e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.548488 5333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c15d9fe0c3724bfbad2156748acd468e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.555673 5391 tablet_bootstrap.cc:654] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:59.560509 5262 tablet_service.cc:1467] Processing CreateTablet for tablet 520249ed5be343dcb30c86d3396491a4 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260929 14:10:59.562268 5340 tablet_service.cc:1467] Processing CreateTablet for tablet 44e89147efa646d3b112a40f53a086a3 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260929 14:10:59.562676 5187 tablet_service.cc:1467] Processing CreateTablet for tablet c15d9fe0c3724bfbad2156748acd468e (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260929 14:10:59.564311 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44e89147efa646d3b112a40f53a086a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.564477 5187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c15d9fe0c3724bfbad2156748acd468e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 14:10:59.565069 5147 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260929 14:10:59.566046 5184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e55a9c8ae48e44c59fee506e539b3f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.568320 5333 tablet_service.cc:1467] Processing CreateTablet for tablet e61c0e0d757e43f19868f329ee0b8bec (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260929 14:10:59.569661 5333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61c0e0d757e43f19868f329ee0b8bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.574842 5257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e50b67ada64545a6b7525d4aba317bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.575730 5332 tablet_service.cc:1467] Processing CreateTablet for tablet 22fb3fae8e7947989abe31ce58000d58 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260929 14:10:59.577039 5332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22fb3fae8e7947989abe31ce58000d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.579380 5262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520249ed5be343dcb30c86d3396491a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.579532 5336 tablet_service.cc:1467] Processing CreateTablet for tablet a3673029b370458db49bab90bbd0bd82 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260929 14:10:59.580811 5336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3673029b370458db49bab90bbd0bd82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.583348 5338 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4a096fb48c46dd95a1d74d34b6e670 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260929 14:10:59.584527 5340 tablet_service.cc:1467] Processing CreateTablet for tablet 7b630b7184c5456bad1fbd45514e52ba (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260929 14:10:59.584761 5333 tablet_service.cc:1467] Processing CreateTablet for tablet 2282d5a97d5a49b9a71bc53d08a7cb65 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260929 14:10:59.584551 5338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4a096fb48c46dd95a1d74d34b6e670. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.586476 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b630b7184c5456bad1fbd45514e52ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.591885 5333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2282d5a97d5a49b9a71bc53d08a7cb65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.594094 5265 tablet_service.cc:1467] Processing CreateTablet for tablet 678eb0db533d4bea872808b6fb39b764 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260929 14:10:59.595398 5265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 678eb0db533d4bea872808b6fb39b764. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.600080 5341 tablet_service.cc:1467] Processing CreateTablet for tablet 08b2fdc7f66741049e11f24a1866c63f (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260929 14:10:59.601800 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfe03c3eac5d42c280945fcb723aabc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.614781 5182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 465f57e1a1654ed1b40ce88c27fa52ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.621856 5339 tablet_service.cc:1467] Processing CreateTablet for tablet e55a9c8ae48e44c59fee506e539b3f90 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260929 14:10:59.642233 5188 tablet_service.cc:1467] Processing CreateTablet for tablet 44e89147efa646d3b112a40f53a086a3 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260929 14:10:59.645041 5188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44e89147efa646d3b112a40f53a086a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.646333 5185 tablet_service.cc:1467] Processing CreateTablet for tablet 22fb3fae8e7947989abe31ce58000d58 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260929 14:10:59.647773 5185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22fb3fae8e7947989abe31ce58000d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.647818 5392 tablet_bootstrap.cc:654] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:59.653055 5341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b2fdc7f66741049e11f24a1866c63f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.655627 5339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e55a9c8ae48e44c59fee506e539b3f90. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 14:10:59.662923 5083 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 64312.7us
I20260929 14:10:59.668789 5338 tablet_service.cc:1467] Processing CreateTablet for tablet 321633daf3914178a03ee03e5c0159d9 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260929 14:10:59.671044 5338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321633daf3914178a03ee03e5c0159d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.676308 5334 tablet_service.cc:1467] Processing CreateTablet for tablet 28e681a7ebc549789ec0a6f1a75c45d9 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260929 14:10:59.680437 5333 tablet_service.cc:1467] Processing CreateTablet for tablet 9cd4876c229b4b768fe4180899274e70 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260929 14:10:59.680538 5187 tablet_service.cc:1467] Processing CreateTablet for tablet a3673029b370458db49bab90bbd0bd82 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260929 14:10:59.683844 5393 tablet_bootstrap.cc:654] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:59.642238 5186 tablet_service.cc:1467] Processing CreateTablet for tablet e61c0e0d757e43f19868f329ee0b8bec (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260929 14:10:59.712093 5186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61c0e0d757e43f19868f329ee0b8bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.720199 5187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3673029b370458db49bab90bbd0bd82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.718040 5334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e681a7ebc549789ec0a6f1a75c45d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.721400 5260 tablet_service.cc:1467] Processing CreateTablet for tablet e55a9c8ae48e44c59fee506e539b3f90 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260929 14:10:59.725690 5260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e55a9c8ae48e44c59fee506e539b3f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.725760 5340 tablet_service.cc:1467] Processing CreateTablet for tablet a3e77685e3424d0da5706ab4f0581529 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260929 14:10:59.728699 5258 tablet_service.cc:1467] Processing CreateTablet for tablet 465f57e1a1654ed1b40ce88c27fa52ac (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260929 14:10:59.729912 5258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 465f57e1a1654ed1b40ce88c27fa52ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.725840 5190 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4a096fb48c46dd95a1d74d34b6e670 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260929 14:10:59.733584 5190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4a096fb48c46dd95a1d74d34b6e670. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.740232 5183 tablet_service.cc:1467] Processing CreateTablet for tablet 2282d5a97d5a49b9a71bc53d08a7cb65 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260929 14:10:59.745960 5183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2282d5a97d5a49b9a71bc53d08a7cb65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.742693 5189 tablet_service.cc:1467] Processing CreateTablet for tablet 08b2fdc7f66741049e11f24a1866c63f (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260929 14:10:59.755514 5189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b2fdc7f66741049e11f24a1866c63f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.757841 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e77685e3424d0da5706ab4f0581529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.761797 5226 heartbeater.cc:503] Master 127.3.242.126:41327 was elected leader, sending a full tablet report...
I20260929 14:10:59.764842 5191 tablet_service.cc:1467] Processing CreateTablet for tablet 7b630b7184c5456bad1fbd45514e52ba (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260929 14:10:59.771440 5333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cd4876c229b4b768fe4180899274e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.774375 5259 tablet_service.cc:1467] Processing CreateTablet for tablet 44e89147efa646d3b112a40f53a086a3 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260929 14:10:59.774892 5391 tablet_bootstrap.cc:492] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:10:59.775485 5391 ts_tablet_manager.cc:1397] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.369s user 0.031s sys 0.006s
I20260929 14:10:59.775756 5259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44e89147efa646d3b112a40f53a086a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.778770 5391 raft_consensus.cc:348] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:10:59.780893 5392 tablet_bootstrap.cc:492] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:10:59.790572 5392 ts_tablet_manager.cc:1397] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.332s user 0.018s sys 0.000s
I20260929 14:10:59.785930 5336 tablet_service.cc:1467] Processing CreateTablet for tablet 5e18ad4fc5524f4687d35e8b2dd03193 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260929 14:10:59.788440 5332 tablet_service.cc:1467] Processing CreateTablet for tablet a38ce5b8e5024c0ab37787aaf2b357c0 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260929 14:10:59.763360 5261 tablet_service.cc:1467] Processing CreateTablet for tablet c15d9fe0c3724bfbad2156748acd468e (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260929 14:10:59.794811 5392 raft_consensus.cc:348] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } }
I20260929 14:10:59.806102 5392 raft_consensus.cc:374] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:59.806488 5392 raft_consensus.cc:729] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:10:59.805872 5332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a38ce5b8e5024c0ab37787aaf2b357c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.809849 5336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e18ad4fc5524f4687d35e8b2dd03193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.798156 5335 tablet_service.cc:1467] Processing CreateTablet for tablet 6978e56e8f49441795cc7f6b6a8f4cfe (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260929 14:10:59.811051 5393 tablet_bootstrap.cc:492] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:10:59.811560 5335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6978e56e8f49441795cc7f6b6a8f4cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.811643 5393 ts_tablet_manager.cc:1397] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.282s user 0.011s sys 0.006s
I20260929 14:10:59.812217 5262 tablet_service.cc:1467] Processing CreateTablet for tablet e61c0e0d757e43f19868f329ee0b8bec (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260929 14:10:59.801612 5338 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9a6f1c02ae4024813fda5af5c665da (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260929 14:10:59.814303 5338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9a6f1c02ae4024813fda5af5c665da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.814209 5337 tablet_service.cc:1467] Processing CreateTablet for tablet 94e36a01dad24c1193b628e9200d3c2d (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260929 14:10:59.814985 5191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b630b7184c5456bad1fbd45514e52ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.815407 5337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e36a01dad24c1193b628e9200d3c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.816865 5391 raft_consensus.cc:374] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:59.817373 5391 raft_consensus.cc:729] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:10:59.818473 5391 consensus_queue.cc:260] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:10:59.795993 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c15d9fe0c3724bfbad2156748acd468e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.826211 5262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e61c0e0d757e43f19868f329ee0b8bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.809790 5392 consensus_queue.cc:260] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } }
I20260929 14:10:59.846393 5263 tablet_service.cc:1467] Processing CreateTablet for tablet 22fb3fae8e7947989abe31ce58000d58 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260929 14:10:59.847920 5263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22fb3fae8e7947989abe31ce58000d58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.853754 5301 heartbeater.cc:503] Master 127.3.242.126:41327 was elected leader, sending a full tablet report...
I20260929 14:10:59.857395 5257 tablet_service.cc:1467] Processing CreateTablet for tablet a3673029b370458db49bab90bbd0bd82 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260929 14:10:59.864605 5266 tablet_service.cc:1467] Processing CreateTablet for tablet 0e4a096fb48c46dd95a1d74d34b6e670 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260929 14:10:59.869633 5257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3673029b370458db49bab90bbd0bd82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.837041 5184 tablet_service.cc:1467] Processing CreateTablet for tablet 321633daf3914178a03ee03e5c0159d9 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260929 14:10:59.870637 5184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321633daf3914178a03ee03e5c0159d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.870805 5341 tablet_service.cc:1467] Processing CreateTablet for tablet 3565847d030f4cc49b113969f438c801 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260929 14:10:59.872116 5341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3565847d030f4cc49b113969f438c801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.879050 5187 tablet_service.cc:1467] Processing CreateTablet for tablet 28e681a7ebc549789ec0a6f1a75c45d9 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260929 14:10:59.879501 5393 raft_consensus.cc:348] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:10:59.880368 5187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e681a7ebc549789ec0a6f1a75c45d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.852548 5182 tablet_service.cc:1467] Processing CreateTablet for tablet 9cd4876c229b4b768fe4180899274e70 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260929 14:10:59.885036 5393 raft_consensus.cc:374] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:10:59.893626 5182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cd4876c229b4b768fe4180899274e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.894586 5391 ts_tablet_manager.cc:1428] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.119s user 0.013s sys 0.000s
I20260929 14:10:59.894971 5258 tablet_service.cc:1467] Processing CreateTablet for tablet 08b2fdc7f66741049e11f24a1866c63f (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260929 14:10:59.895527 5391 tablet_bootstrap.cc:492] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:10:59.903007 5260 tablet_service.cc:1467] Processing CreateTablet for tablet 321633daf3914178a03ee03e5c0159d9 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260929 14:10:59.904661 5264 tablet_service.cc:1467] Processing CreateTablet for tablet 7b630b7184c5456bad1fbd45514e52ba (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260929 14:10:59.905457 5185 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9a6f1c02ae4024813fda5af5c665da (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260929 14:10:59.907613 5185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9a6f1c02ae4024813fda5af5c665da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.916832 5333 tablet_service.cc:1467] Processing CreateTablet for tablet 625b1b3f659041f8b54de3499628ebc4 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260929 14:10:59.918125 5333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625b1b3f659041f8b54de3499628ebc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.929555 5182 tablet_service.cc:1467] Processing CreateTablet for tablet 6978e56e8f49441795cc7f6b6a8f4cfe (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260929 14:10:59.912150 5189 tablet_service.cc:1467] Processing CreateTablet for tablet a38ce5b8e5024c0ab37787aaf2b357c0 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260929 14:10:59.912835 5190 tablet_service.cc:1467] Processing CreateTablet for tablet a3e77685e3424d0da5706ab4f0581529 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260929 14:10:59.913950 5265 tablet_service.cc:1467] Processing CreateTablet for tablet 2282d5a97d5a49b9a71bc53d08a7cb65 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260929 14:10:59.933722 5182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6978e56e8f49441795cc7f6b6a8f4cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.915239 5186 tablet_service.cc:1467] Processing CreateTablet for tablet 5e18ad4fc5524f4687d35e8b2dd03193 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260929 14:10:59.935444 5186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e18ad4fc5524f4687d35e8b2dd03193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.916674 5340 tablet_service.cc:1467] Processing CreateTablet for tablet e9e4f58231414bb5a9b16e093027d0c4 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260929 14:10:59.936771 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e4f58231414bb5a9b16e093027d0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.937744 5190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e77685e3424d0da5706ab4f0581529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.891407 5393 raft_consensus.cc:729] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:10:59.941670 5189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a38ce5b8e5024c0ab37787aaf2b357c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.930401 5392 ts_tablet_manager.cc:1428] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.139s user 0.008s sys 0.006s
I20260929 14:10:59.942276 5393 consensus_queue.cc:260] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:10:59.943722 5392 tablet_bootstrap.cc:492] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:10:59.891094 5266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4a096fb48c46dd95a1d74d34b6e670. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.945780 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b630b7184c5456bad1fbd45514e52ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.949492 5260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 321633daf3914178a03ee03e5c0159d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.930382 5334 tablet_service.cc:1467] Processing CreateTablet for tablet f7507b3dbd59424f8887991236a4a49c (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260929 14:10:59.953526 5258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b2fdc7f66741049e11f24a1866c63f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.954188 5334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7507b3dbd59424f8887991236a4a49c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.954393 5191 tablet_service.cc:1467] Processing CreateTablet for tablet 94e36a01dad24c1193b628e9200d3c2d (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260929 14:10:59.923720 5339 tablet_service.cc:1467] Processing CreateTablet for tablet 2818f47d8c6b41c1ab134b96096cacdc (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260929 14:10:59.955587 5188 tablet_service.cc:1467] Processing CreateTablet for tablet 3565847d030f4cc49b113969f438c801 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260929 14:10:59.956571 5339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2818f47d8c6b41c1ab134b96096cacdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.908998 5391 tablet_bootstrap.cc:654] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:10:59.969520 5265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2282d5a97d5a49b9a71bc53d08a7cb65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.972631 5376 heartbeater.cc:503] Master 127.3.242.126:41327 was elected leader, sending a full tablet report...
I20260929 14:10:59.955600 5191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e36a01dad24c1193b628e9200d3c2d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 14:10:59.977895 5227 tablet.cc:2369] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 14:10:59.978360 5332 tablet_service.cc:1467] Processing CreateTablet for tablet 19257f7a5d364214b38d26c5f9b6d190 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260929 14:10:59.991014 5332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19257f7a5d364214b38d26c5f9b6d190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.993826 5393 ts_tablet_manager.cc:1428] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.182s user 0.010s sys 0.000s
I20260929 14:10:59.993525 5259 tablet_service.cc:1467] Processing CreateTablet for tablet 9cd4876c229b4b768fe4180899274e70 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260929 14:10:59.994889 5259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cd4876c229b4b768fe4180899274e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.994932 5393 tablet_bootstrap.cc:492] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:10:59.995766 5183 tablet_service.cc:1467] Processing CreateTablet for tablet e9e4f58231414bb5a9b16e093027d0c4 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260929 14:10:59.997105 5183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e4f58231414bb5a9b16e093027d0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:10:59.993518 5336 tablet_service.cc:1467] Processing CreateTablet for tablet dbca949bc56f4fb19a19b93566383d8d (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260929 14:11:00.009464 5188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3565847d030f4cc49b113969f438c801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.009320 5258 tablet_service.cc:1467] Processing CreateTablet for tablet 5e18ad4fc5524f4687d35e8b2dd03193 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260929 14:11:00.010416 5261 tablet_service.cc:1467] Processing CreateTablet for tablet a38ce5b8e5024c0ab37787aaf2b357c0 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260929 14:11:00.011248 5260 tablet_service.cc:1467] Processing CreateTablet for tablet 28e681a7ebc549789ec0a6f1a75c45d9 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260929 14:11:00.013082 5336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbca949bc56f4fb19a19b93566383d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.017786 5392 tablet_bootstrap.cc:654] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.022209 5337 tablet_service.cc:1467] Processing CreateTablet for tablet 929c62a6106e42d08a13689e62dce3b1 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260929 14:11:00.023404 5337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 929c62a6106e42d08a13689e62dce3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.025498 5260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e681a7ebc549789ec0a6f1a75c45d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.029497 5258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e18ad4fc5524f4687d35e8b2dd03193. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.030265 5263 tablet_service.cc:1467] Processing CreateTablet for tablet a3e77685e3424d0da5706ab4f0581529 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260929 14:11:00.031559 5263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e77685e3424d0da5706ab4f0581529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.032851 5338 tablet_service.cc:1467] Processing CreateTablet for tablet f9ebe16909c64202bcf70eb0425efdca (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260929 14:11:00.033519 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a38ce5b8e5024c0ab37787aaf2b357c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.034106 5338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ebe16909c64202bcf70eb0425efdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.031435 5186 tablet_service.cc:1467] Processing CreateTablet for tablet 625b1b3f659041f8b54de3499628ebc4 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260929 14:11:00.036860 5340 tablet_service.cc:1467] Processing CreateTablet for tablet 62fed7cb04b44fca9312372e511f1c3e (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260929 14:11:00.037936 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62fed7cb04b44fca9312372e511f1c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.043679 5186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625b1b3f659041f8b54de3499628ebc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.046099 5190 tablet_service.cc:1467] Processing CreateTablet for tablet 2818f47d8c6b41c1ab134b96096cacdc (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260929 14:11:00.053627 5264 tablet_service.cc:1467] Processing CreateTablet for tablet 6978e56e8f49441795cc7f6b6a8f4cfe (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260929 14:11:00.055029 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6978e56e8f49441795cc7f6b6a8f4cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.055380 5257 tablet_service.cc:1467] Processing CreateTablet for tablet 94e36a01dad24c1193b628e9200d3c2d (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260929 14:11:00.056565 5257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e36a01dad24c1193b628e9200d3c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.057410 5190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2818f47d8c6b41c1ab134b96096cacdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.074786 5391 tablet_bootstrap.cc:492] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:00.075265 5397 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:00.075742 5397 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.078069 5397 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:00.079531 5340 tablet_service.cc:1467] Processing CreateTablet for tablet e5acb0315ba44b5a9d145119868e7168 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260929 14:11:00.080940 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5acb0315ba44b5a9d145119868e7168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.082011 5333 tablet_service.cc:1467] Processing CreateTablet for tablet ae31abec82ff4f40b695ccb19f289dbc (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260929 14:11:00.083824 5333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae31abec82ff4f40b695ccb19f289dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.075294 5391 ts_tablet_manager.cc:1397] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.180s user 0.010s sys 0.000s
I20260929 14:11:00.110687 5393 tablet_bootstrap.cc:654] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.113197 5182 tablet_service.cc:1467] Processing CreateTablet for tablet f9ebe16909c64202bcf70eb0425efdca (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260929 14:11:00.114629 5182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ebe16909c64202bcf70eb0425efdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.038266 5262 tablet_service.cc:1467] Processing CreateTablet for tablet 7e9a6f1c02ae4024813fda5af5c665da (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260929 14:11:00.136015 5340 tablet_service.cc:1467] Processing CreateTablet for tablet 4040985909414481ad0d1491249e8e67 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260929 14:11:00.137248 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4040985909414481ad0d1491249e8e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.115986 5183 tablet_service.cc:1467] Processing CreateTablet for tablet 929c62a6106e42d08a13689e62dce3b1 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260929 14:11:00.147610 5183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 929c62a6106e42d08a13689e62dce3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.155691 5333 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3538be0f624dd79d0e667313b22d02 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260929 14:11:00.159916 5333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3538be0f624dd79d0e667313b22d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.177487 5189 tablet_service.cc:1467] Processing CreateTablet for tablet dbca949bc56f4fb19a19b93566383d8d (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260929 14:11:00.178752 5189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbca949bc56f4fb19a19b93566383d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.179684 5392 tablet_bootstrap.cc:492] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:00.180152 5392 ts_tablet_manager.cc:1397] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.237s user 0.028s sys 0.000s
I20260929 14:11:00.183986 5392 raft_consensus.cc:348] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.185331 5392 raft_consensus.cc:374] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.185899 5392 raft_consensus.cc:729] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.186672 5399 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:00.187245 5399 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.120697 5184 tablet_service.cc:1467] Processing CreateTablet for tablet f7507b3dbd59424f8887991236a4a49c (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260929 14:11:00.193481 5340 tablet_service.cc:1467] Processing CreateTablet for tablet d4dd0a13d8aa4e6bbd3633c047984f67 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260929 14:11:00.200403 5340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4dd0a13d8aa4e6bbd3633c047984f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.199213 5262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e9a6f1c02ae4024813fda5af5c665da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.191900 5399 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:00.231901 5182 tablet_service.cc:1467] Processing CreateTablet for tablet 4040985909414481ad0d1491249e8e67 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260929 14:11:00.233206 5182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4040985909414481ad0d1491249e8e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.233558 5332 tablet_service.cc:1467] Processing CreateTablet for tablet 6239036de00f41c1b2ec5537a1ddc982 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260929 14:11:00.234846 5332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6239036de00f41c1b2ec5537a1ddc982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.198841 5266 tablet_service.cc:1467] Processing CreateTablet for tablet 3565847d030f4cc49b113969f438c801 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260929 14:11:00.189432 5336 tablet_service.cc:1467] Processing CreateTablet for tablet 4300e83384e9458bb280a4b1e3131932 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260929 14:11:00.251957 5398 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:00.252449 5398 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } }
I20260929 14:11:00.253537 5266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3565847d030f4cc49b113969f438c801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.206543 5337 tablet_service.cc:1467] Processing CreateTablet for tablet 632bdccb08444a4d89ecc1cfdbab98b2 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
W20260929 14:11:00.268400 5086 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 49100.8us
I20260929 14:11:00.271656 5261 tablet_service.cc:1467] Processing CreateTablet for tablet e9e4f58231414bb5a9b16e093027d0c4 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260929 14:11:00.272941 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e4f58231414bb5a9b16e093027d0c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.272774 5263 tablet_service.cc:1467] Processing CreateTablet for tablet 625b1b3f659041f8b54de3499628ebc4 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260929 14:11:00.273972 5263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 625b1b3f659041f8b54de3499628ebc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.294410 5393 tablet_bootstrap.cc:492] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:00.294952 5393 ts_tablet_manager.cc:1397] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.300s user 0.023s sys 0.003s
I20260929 14:11:00.295006 5182 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3538be0f624dd79d0e667313b22d02 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260929 14:11:00.263839 5398 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:00.296294 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:00.296860 5265 tablet_service.cc:1467] Processing CreateTablet for tablet 2818f47d8c6b41c1ab134b96096cacdc (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260929 14:11:00.297173 5333 tablet_service.cc:1467] Processing CreateTablet for tablet 01b3432723044cbab3d1437e32a2efda (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260929 14:11:00.298134 5265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2818f47d8c6b41c1ab134b96096cacdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.298357 5333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b3432723044cbab3d1437e32a2efda. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 14:11:00.298435 5164 leader_election.cc:341] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.211673 5392 consensus_queue.cc:260] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.211903 5187 tablet_service.cc:1467] Processing CreateTablet for tablet 19257f7a5d364214b38d26c5f9b6d190 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260929 14:11:00.231907 5186 tablet_service.cc:1467] Processing CreateTablet for tablet ae31abec82ff4f40b695ccb19f289dbc (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260929 14:11:00.325599 5336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4300e83384e9458bb280a4b1e3131932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.349114 5185 tablet_service.cc:1467] Processing CreateTablet for tablet 62fed7cb04b44fca9312372e511f1c3e (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260929 14:11:00.353977 5392 ts_tablet_manager.cc:1428] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.174s user 0.009s sys 0.000s
I20260929 14:11:00.354903 5392 tablet_bootstrap.cc:492] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:00.345172 5391 raft_consensus.cc:348] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } }
I20260929 14:11:00.358573 5191 tablet_service.cc:1467] Processing CreateTablet for tablet e5acb0315ba44b5a9d145119868e7168 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260929 14:11:00.360304 5260 tablet_service.cc:1467] Processing CreateTablet for tablet f7507b3dbd59424f8887991236a4a49c (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260929 14:11:00.359254 5182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3538be0f624dd79d0e667313b22d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.364795 5264 tablet_service.cc:1467] Processing CreateTablet for tablet 19257f7a5d364214b38d26c5f9b6d190 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260929 14:11:00.366113 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19257f7a5d364214b38d26c5f9b6d190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.369560 5260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7507b3dbd59424f8887991236a4a49c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.379017 5341 tablet_service.cc:1467] Processing CreateTablet for tablet f2610284ac4749c5bb443c221df8ff92 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260929 14:11:00.382067 5334 tablet_service.cc:1467] Processing CreateTablet for tablet 91e285df817143ec9947058346e02b90 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260929 14:11:00.389459 5337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 632bdccb08444a4d89ecc1cfdbab98b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.390103 5334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e285df817143ec9947058346e02b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.392514 5391 raft_consensus.cc:374] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.392871 5391 raft_consensus.cc:729] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.393625 5391 consensus_queue.cc:260] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } }
I20260929 14:11:00.395711 5391 ts_tablet_manager.cc:1428] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.302s user 0.001s sys 0.004s
I20260929 14:11:00.397529 5341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2610284ac4749c5bb443c221df8ff92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.398213 5190 tablet_service.cc:1467] Processing CreateTablet for tablet 4300e83384e9458bb280a4b1e3131932 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260929 14:11:00.399377 5190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4300e83384e9458bb280a4b1e3131932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.401825 5187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19257f7a5d364214b38d26c5f9b6d190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.402379 5185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62fed7cb04b44fca9312372e511f1c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.405391 5186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae31abec82ff4f40b695ccb19f289dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.405985 5184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7507b3dbd59424f8887991236a4a49c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.407023 5188 tablet_service.cc:1467] Processing CreateTablet for tablet 632bdccb08444a4d89ecc1cfdbab98b2 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260929 14:11:00.408239 5188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 632bdccb08444a4d89ecc1cfdbab98b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.410300 5191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5acb0315ba44b5a9d145119868e7168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.340556 5393 raft_consensus.cc:348] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.413789 5393 raft_consensus.cc:374] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.414098 5393 raft_consensus.cc:729] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.414827 5393 consensus_queue.cc:260] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.404987 5260 tablet_service.cc:1467] Processing CreateTablet for tablet dbca949bc56f4fb19a19b93566383d8d (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260929 14:11:00.424736 5260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbca949bc56f4fb19a19b93566383d8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.426249 5264 tablet_service.cc:1467] Processing CreateTablet for tablet 929c62a6106e42d08a13689e62dce3b1 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260929 14:11:00.427470 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 929c62a6106e42d08a13689e62dce3b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.431419 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:00.434150 5314 leader_election.cc:341] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.435506 5261 tablet_service.cc:1467] Processing CreateTablet for tablet 62fed7cb04b44fca9312372e511f1c3e (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260929 14:11:00.438009 5182 tablet_service.cc:1467] Processing CreateTablet for tablet d4dd0a13d8aa4e6bbd3633c047984f67 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260929 14:11:00.438630 5257 tablet_service.cc:1467] Processing CreateTablet for tablet e5acb0315ba44b5a9d145119868e7168 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260929 14:11:00.439234 5182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4dd0a13d8aa4e6bbd3633c047984f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.440366 5391 tablet_bootstrap.cc:492] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:00.443722 5399 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:00.444186 5399 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.446127 5399 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:00.452333 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:00.452714 5393 ts_tablet_manager.cc:1428] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.157s user 0.004s sys 0.005s
W20260929 14:11:00.454877 5314 leader_election.cc:341] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.457172 5393 tablet_bootstrap.cc:492] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:00.460758 5191 tablet_service.cc:1467] Processing CreateTablet for tablet 01b3432723044cbab3d1437e32a2efda (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260929 14:11:00.464251 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62fed7cb04b44fca9312372e511f1c3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.467306 5191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b3432723044cbab3d1437e32a2efda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.467972 5391 tablet_bootstrap.cc:654] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.469240 5257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5acb0315ba44b5a9d145119868e7168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.470103 5266 tablet_service.cc:1467] Processing CreateTablet for tablet f9ebe16909c64202bcf70eb0425efdca (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260929 14:11:00.453006 5189 tablet_service.cc:1467] Processing CreateTablet for tablet 6239036de00f41c1b2ec5537a1ddc982 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260929 14:11:00.494338 5189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6239036de00f41c1b2ec5537a1ddc982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.498019 5392 tablet_bootstrap.cc:654] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.498251 5184 tablet_service.cc:1467] Processing CreateTablet for tablet f2610284ac4749c5bb443c221df8ff92 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260929 14:11:00.499478 5184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2610284ac4749c5bb443c221df8ff92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.477859 5183 tablet_service.cc:1467] Processing CreateTablet for tablet 91e285df817143ec9947058346e02b90 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260929 14:11:00.459542 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:00.484664 5393 tablet_bootstrap.cc:654] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.490307 5258 tablet_service.cc:1467] Processing CreateTablet for tablet 0c3538be0f624dd79d0e667313b22d02 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260929 14:11:00.491496 5263 tablet_service.cc:1467] Processing CreateTablet for tablet ae31abec82ff4f40b695ccb19f289dbc (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260929 14:11:00.492667 5266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9ebe16909c64202bcf70eb0425efdca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.470548 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:00.501988 5259 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:50768 (request call id 30) took 1056 ms. Trace:
W20260929 14:11:00.509053 5165 leader_election.cc:341] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.509582 5260 tablet_service.cc:1467] Processing CreateTablet for tablet 4300e83384e9458bb280a4b1e3131932 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260929 14:11:00.514691 5165 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:00.524605 5239 leader_election.cc:341] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.525187 5397 raft_consensus.cc:2726] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:00.512861 5259 rpcz_store.cc:276] 0929 14:10:59.446103 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:10:59.992441 (+546338us) service_pool.cc:227] Handling call
0929 14:10:59.994421 (+ 1980us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0929 14:10:59.994519 (+ 98us) ts_tablet_manager.cc:676] Creating new metadata...
0929 14:11:00.326558 (+332039us) tablet_metadata.cc:641] Metadata flushed
0929 14:11:00.501488 (+174930us) tablet_replica.cc:205] Creating consensus instance
0929 14:11:00.501931 (+ 443us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:00.528450 5183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e285df817143ec9947058346e02b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.535581 5258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c3538be0f624dd79d0e667313b22d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.515549 5262 tablet_service.cc:1467] Processing CreateTablet for tablet 4040985909414481ad0d1491249e8e67 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260929 14:11:00.528267 5259 tablet_service.cc:1467] Processing CreateTablet for tablet 632bdccb08444a4d89ecc1cfdbab98b2 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260929 14:11:00.544780 5391 tablet_bootstrap.cc:492] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:00.545307 5391 ts_tablet_manager.cc:1397] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.105s user 0.009s sys 0.007s
I20260929 14:11:00.547001 5263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae31abec82ff4f40b695ccb19f289dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.547890 5265 tablet_service.cc:1467] Processing CreateTablet for tablet d4dd0a13d8aa4e6bbd3633c047984f67 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260929 14:11:00.548652 5261 tablet_service.cc:1467] Processing CreateTablet for tablet 6239036de00f41c1b2ec5537a1ddc982 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260929 14:11:00.549059 5265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4dd0a13d8aa4e6bbd3633c047984f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.547921 5391 raft_consensus.cc:348] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.553859 5391 raft_consensus.cc:374] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.554211 5391 raft_consensus.cc:729] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.555083 5260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4300e83384e9458bb280a4b1e3131932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.554960 5391 consensus_queue.cc:260] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.557744 5262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4040985909414481ad0d1491249e8e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.559778 5264 tablet_service.cc:1467] Processing CreateTablet for tablet 01b3432723044cbab3d1437e32a2efda (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260929 14:11:00.561192 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6239036de00f41c1b2ec5537a1ddc982. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.560388 5257 tablet_service.cc:1467] Processing CreateTablet for tablet f2610284ac4749c5bb443c221df8ff92 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260929 14:11:00.562263 5257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2610284ac4749c5bb443c221df8ff92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.560778 5391 ts_tablet_manager.cc:1428] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.015s user 0.004s sys 0.003s
I20260929 14:11:00.563551 5264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01b3432723044cbab3d1437e32a2efda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.574330 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:00.575083 5201 raft_consensus.cc:2454] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:00.576464 5242 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:00.577625 5398 raft_consensus.cc:2781] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:00.577965 5398 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:00.578305 5398 raft_consensus.cc:3037] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:00.581674 5261 tablet_service.cc:1467] Processing CreateTablet for tablet 91e285df817143ec9947058346e02b90 (DEFAULT_TABLE table=test-workload [id=7efbef67b40f466dae10d9b6ed991948]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260929 14:11:00.582832 5261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e285df817143ec9947058346e02b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.584790 5398 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } }
I20260929 14:11:00.588020 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:00.588982 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:00.589152 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:00.589993 5199 raft_consensus.cc:3037] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:11:00.591013 5239 leader_election.cc:341] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
W20260929 14:11:00.592504 5316 leader_election.cc:341] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.591130 5398 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:00.592965 5316 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:00.594152 5399 raft_consensus.cc:2726] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:00.595093 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:00.597543 5147 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260929 14:11:00.597743 5316 leader_election.cc:341] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.598197 5316 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:00.599395 5399 raft_consensus.cc:2726] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:00.600513 5391 tablet_bootstrap.cc:492] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:00.604732 5259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 632bdccb08444a4d89ecc1cfdbab98b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:00.605047 5392 tablet_bootstrap.cc:492] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:00.605528 5392 ts_tablet_manager.cc:1397] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.251s user 0.010s sys 0.013s
I20260929 14:11:00.607945 5391 tablet_bootstrap.cc:654] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.608121 5392 raft_consensus.cc:348] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.608843 5392 raft_consensus.cc:374] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.609104 5392 raft_consensus.cc:729] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.612998 5392 consensus_queue.cc:260] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.615217 5392 ts_tablet_manager.cc:1428] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260929 14:11:00.621958 5392 tablet_bootstrap.cc:492] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:00.627496 5393 tablet_bootstrap.cc:492] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:00.627996 5393 ts_tablet_manager.cc:1397] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.174s user 0.009s sys 0.008s
I20260929 14:11:00.629384 5199 raft_consensus.cc:2454] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:00.630453 5242 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:00.631148 5398 raft_consensus.cc:2781] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:00.630903 5393 raft_consensus.cc:348] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.631664 5393 raft_consensus.cc:374] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.633035 5393 raft_consensus.cc:729] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.638490 5397 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:00.638954 5397 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.640941 5397 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:00.645752 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:00.641430 5393 consensus_queue.cc:260] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.647856 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:00.650290 5164 leader_election.cc:341] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
W20260929 14:11:00.651341 5165 leader_election.cc:341] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.651927 5165 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:00.654614 5397 raft_consensus.cc:2726] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:00.655850 5392 tablet_bootstrap.cc:654] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.654445 5398 raft_consensus.cc:686] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:00.650326 5393 ts_tablet_manager.cc:1428] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.022s user 0.008s sys 0.000s
I20260929 14:11:00.661164 5398 consensus_queue.cc:237] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } }
I20260929 14:11:00.662945 5419 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:00.670022 5419 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.673243 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:00.674841 5242 leader_election.cc:341] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.668479 5393 tablet_bootstrap.cc:492] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:00.682114 5419 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:00.682719 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:00.683899 5239 leader_election.cc:341] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.684327 5239 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:00.685194 5393 tablet_bootstrap.cc:654] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.685561 5419 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:00.686779 5392 tablet_bootstrap.cc:492] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:00.687242 5392 ts_tablet_manager.cc:1397] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.065s user 0.010s sys 0.000s
I20260929 14:11:00.689543 5392 raft_consensus.cc:348] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.690259 5392 raft_consensus.cc:374] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.690490 5392 raft_consensus.cc:729] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.691072 5392 consensus_queue.cc:260] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.685050 5391 tablet_bootstrap.cc:492] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:00.692241 5391 ts_tablet_manager.cc:1397] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.092s user 0.013s sys 0.007s
I20260929 14:11:00.692950 5393 tablet_bootstrap.cc:492] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:00.693761 5393 ts_tablet_manager.cc:1397] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.028s user 0.012s sys 0.000s
I20260929 14:11:00.694330 5391 raft_consensus.cc:348] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.694854 5391 raft_consensus.cc:374] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.695087 5391 raft_consensus.cc:729] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.695676 5391 consensus_queue.cc:260] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.697590 5391 ts_tablet_manager.cc:1428] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:11:00.697654 5393 raft_consensus.cc:348] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.698808 5392 ts_tablet_manager.cc:1428] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260929 14:11:00.698773 5393 raft_consensus.cc:374] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.699398 5393 raft_consensus.cc:729] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.700356 5393 consensus_queue.cc:260] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.699712 5392 tablet_bootstrap.cc:492] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:00.702922 5393 ts_tablet_manager.cc:1428] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260929 14:11:00.704232 5393 tablet_bootstrap.cc:492] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:00.706321 5398 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:00.706781 5398 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.707223 5098 catalog_manager.cc:5495] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:00.709643 5391 tablet_bootstrap.cc:492] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:00.710004 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:00.710767 5199 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:00.712087 5242 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:00.712984 5419 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:00.713361 5419 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:00.713789 5419 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:00.716472 5391 tablet_bootstrap.cc:654] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.719966 5398 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:00.720976 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:00.713351 5393 tablet_bootstrap.cc:654] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.721649 5351 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:00.725206 5392 tablet_bootstrap.cc:654] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.722271 5419 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.731453 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:00.732038 5199 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:00.732203 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:00.733028 5351 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:00.737859 5392 tablet_bootstrap.cc:492] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:00.738349 5392 ts_tablet_manager.cc:1397] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.000s
I20260929 14:11:00.740551 5199 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:00.740957 5392 raft_consensus.cc:348] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.741708 5392 raft_consensus.cc:374] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.741696 5242 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:00.742146 5392 raft_consensus.cc:729] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.742692 5398 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:00.743119 5398 raft_consensus.cc:686] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:00.743045 5392 consensus_queue.cc:260] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.743932 5398 consensus_queue.cc:237] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.741359 5419 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:00.746663 5351 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:00.754127 5392 ts_tablet_manager.cc:1428] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260929 14:11:00.754148 5393 tablet_bootstrap.cc:492] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:00.755349 5393 ts_tablet_manager.cc:1397] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.051s user 0.011s sys 0.003s
I20260929 14:11:00.755474 5392 tablet_bootstrap.cc:492] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:00.761075 5393 raft_consensus.cc:348] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.761770 5393 raft_consensus.cc:374] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.762075 5393 raft_consensus.cc:729] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.762750 5393 consensus_queue.cc:260] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.764736 5392 tablet_bootstrap.cc:654] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.764925 5391 tablet_bootstrap.cc:492] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:00.766006 5391 ts_tablet_manager.cc:1397] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.057s user 0.032s sys 0.019s
I20260929 14:11:00.768966 5391 raft_consensus.cc:348] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.770001 5391 raft_consensus.cc:374] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.770308 5391 raft_consensus.cc:729] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.771024 5391 consensus_queue.cc:260] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.773062 5391 ts_tablet_manager.cc:1428] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260929 14:11:00.773506 5398 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:00.773988 5398 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.774902 5392 tablet_bootstrap.cc:492] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:00.775352 5392 ts_tablet_manager.cc:1397] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.006s
I20260929 14:11:00.775921 5398 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:00.777040 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:00.777376 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:00.777726 5351 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:00.777841 5393 ts_tablet_manager.cc:1428] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20260929 14:11:00.777931 5392 raft_consensus.cc:348] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.778761 5392 raft_consensus.cc:374] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.779026 5392 raft_consensus.cc:729] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.778975 5239 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:00.779639 5242 leader_election.cc:341] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.779810 5392 consensus_queue.cc:260] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.780990 5419 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:00.781283 5393 tablet_bootstrap.cc:492] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:00.781389 5419 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:00.781955 5419 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:00.783267 5098 catalog_manager.cc:5495] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:00.785049 5391 tablet_bootstrap.cc:492] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:00.787957 5393 tablet_bootstrap.cc:654] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.789969 5419 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.791247 5391 tablet_bootstrap.cc:654] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.782029 5392 ts_tablet_manager.cc:1428] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 14:11:00.792999 5392 tablet_bootstrap.cc:492] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:00.794546 5398 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:00.794989 5398 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.796386 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:00.797075 5398 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:00.797967 5242 leader_election.cc:341] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.798486 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:00.799121 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:00.799703 5199 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
W20260929 14:11:00.799759 5239 leader_election.cc:341] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.800743 5392 tablet_bootstrap.cc:654] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.801715 5419 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:00.802647 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:00.803198 5351 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:00.805437 5242 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:00.808102 5399 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:00.808700 5399 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.809662 5419 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:00.810005 5419 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:00.810335 5419 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:00.811453 5351 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:00.812521 5239 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:00.813580 5398 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:00.814015 5398 raft_consensus.cc:686] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:00.814321 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:00.814776 5398 consensus_queue.cc:237] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:00.816445 5314 leader_election.cc:341] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.820158 5399 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:00.821236 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:00.823464 5316 leader_election.cc:341] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.823881 5316 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:00.816823 5419 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.826905 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:00.827527 5201 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:00.828905 5399 raft_consensus.cc:2726] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:00.830696 5391 tablet_bootstrap.cc:492] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:00.831146 5391 ts_tablet_manager.cc:1397] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.048s user 0.012s sys 0.004s
I20260929 14:11:00.836220 5393 tablet_bootstrap.cc:492] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:00.837065 5393 ts_tablet_manager.cc:1397] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.056s user 0.013s sys 0.015s
I20260929 14:11:00.837849 5201 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:00.838984 5242 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:00.840241 5418 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:00.838939 5391 raft_consensus.cc:348] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.841099 5391 raft_consensus.cc:374] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.841677 5391 raft_consensus.cc:729] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.841755 5393 raft_consensus.cc:348] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.843014 5393 raft_consensus.cc:374] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.843426 5393 raft_consensus.cc:729] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.844612 5391 consensus_queue.cc:260] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.844499 5393 consensus_queue.cc:260] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.847348 5391 ts_tablet_manager.cc:1428] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260929 14:11:00.848291 5391 tablet_bootstrap.cc:492] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:00.849758 5393 ts_tablet_manager.cc:1428] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.012s user 0.007s sys 0.001s
I20260929 14:11:00.850963 5393 tablet_bootstrap.cc:492] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:00.859750 5393 tablet_bootstrap.cc:654] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.864470 5391 tablet_bootstrap.cc:654] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.874305 5391 tablet_bootstrap.cc:492] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:00.874792 5391 ts_tablet_manager.cc:1397] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.027s user 0.007s sys 0.003s
I20260929 14:11:00.877444 5393 tablet_bootstrap.cc:492] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:00.877384 5391 raft_consensus.cc:348] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.878183 5391 raft_consensus.cc:374] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.878326 5393 ts_tablet_manager.cc:1397] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.016s
I20260929 14:11:00.881812 5419 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:00.881893 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:00.882746 5393 raft_consensus.cc:348] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:00.883420 5239 leader_election.cc:341] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.883966 5393 raft_consensus.cc:374] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.884469 5393 raft_consensus.cc:729] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.886138 5418 raft_consensus.cc:686] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:00.885813 5393 consensus_queue.cc:260] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.878492 5391 raft_consensus.cc:729] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.887233 5418 consensus_queue.cc:237] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.887771 5391 consensus_queue.cc:260] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.890249 5391 ts_tablet_manager.cc:1428] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260929 14:11:00.891289 5391 tablet_bootstrap.cc:492] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:00.894237 5393 ts_tablet_manager.cc:1428] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.015s user 0.004s sys 0.005s
I20260929 14:11:00.895716 5393 tablet_bootstrap.cc:492] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:00.897538 5391 tablet_bootstrap.cc:654] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.898170 5099 catalog_manager.cc:5495] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:00.904521 5393 tablet_bootstrap.cc:654] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.917243 5393 tablet_bootstrap.cc:492] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:00.918000 5393 ts_tablet_manager.cc:1397] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.004s
I20260929 14:11:00.928117 5393 raft_consensus.cc:348] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.929198 5393 raft_consensus.cc:374] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.929625 5393 raft_consensus.cc:729] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.930634 5393 consensus_queue.cc:260] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.940976 5393 ts_tablet_manager.cc:1428] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.023s user 0.002s sys 0.005s
I20260929 14:11:00.942374 5393 tablet_bootstrap.cc:492] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:00.945818 5397 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:00.946291 5397 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.948235 5397 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:00.949685 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:00.950372 5351 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:00.951548 5164 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:00.951218 5098 catalog_manager.cc:5495] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:00.953646 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:00.955039 5165 leader_election.cc:341] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.955895 5397 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:00.956274 5397 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:00.956586 5397 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:00.957032 5393 tablet_bootstrap.cc:654] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.956480 5392 tablet_bootstrap.cc:492] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:00.963702 5392 ts_tablet_manager.cc:1397] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.171s user 0.042s sys 0.115s
I20260929 14:11:00.963778 5399 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:00.965478 5397 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.965694 5399 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.964958 5393 tablet_bootstrap.cc:492] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:00.966681 5393 ts_tablet_manager.cc:1397] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.001s
I20260929 14:11:00.967331 5391 tablet_bootstrap.cc:492] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:00.967759 5391 ts_tablet_manager.cc:1397] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.077s user 0.026s sys 0.034s
I20260929 14:11:00.967696 5392 raft_consensus.cc:348] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.968647 5392 raft_consensus.cc:374] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.968906 5392 raft_consensus.cc:729] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.968927 5393 raft_consensus.cc:348] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.969727 5393 raft_consensus.cc:374] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.970046 5393 raft_consensus.cc:729] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.969779 5392 consensus_queue.cc:260] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.970439 5391 raft_consensus.cc:348] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.970678 5393 consensus_queue.cc:260] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:00.971102 5391 raft_consensus.cc:374] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:00.970988 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:00.971398 5391 raft_consensus.cc:729] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:00.969246 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:00.972246 5391 consensus_queue.cc:260] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:00.972987 5314 leader_election.cc:341] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:00.967696 5399 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:00.975109 5392 ts_tablet_manager.cc:1428] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260929 14:11:00.976001 5392 tablet_bootstrap.cc:492] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:00.972316 5201 raft_consensus.cc:2454] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:00.984051 5392 tablet_bootstrap.cc:654] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:00.984056 5316 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:00.985564 5399 raft_consensus.cc:2781] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:00.985991 5399 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:00.986404 5399 raft_consensus.cc:3037] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:00.996429 5392 tablet_bootstrap.cc:492] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:00.996973 5392 ts_tablet_manager.cc:1397] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.011s
I20260929 14:11:00.999064 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:00.999720 5351 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.000161 5392 raft_consensus.cc:348] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.000865 5392 raft_consensus.cc:374] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.001118 5392 raft_consensus.cc:729] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.001984 5392 consensus_queue.cc:260] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.003798 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:01.004280 5392 ts_tablet_manager.cc:1428] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 14:11:01.004644 5275 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.005187 5392 tablet_bootstrap.cc:492] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:01.006054 5418 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:01.006619 5418 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.007007 5351 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:01.008720 5164 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:01.010980 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:01.011735 5392 tablet_bootstrap.cc:654] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.011894 5350 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:01.012833 5418 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:01.014758 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:01.015184 5239 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:01.015410 5201 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:01.016232 5447 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:01.016691 5447 raft_consensus.cc:686] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:01.014362 5397 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:01.017478 5447 consensus_queue.cc:237] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.021732 5418 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:01.023574 5418 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:01.023988 5418 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.024205 5392 tablet_bootstrap.cc:492] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:01.018245 5399 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.024685 5392 ts_tablet_manager.cc:1397] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.008s
I20260929 14:11:01.026679 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:01.027253 5201 raft_consensus.cc:3037] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.027194 5392 raft_consensus.cc:348] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.027938 5392 raft_consensus.cc:374] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.028324 5392 raft_consensus.cc:729] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.029006 5392 consensus_queue.cc:260] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.031098 5392 ts_tablet_manager.cc:1428] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260929 14:11:01.031937 5392 tablet_bootstrap.cc:492] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:01.037360 5201 raft_consensus.cc:2454] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:01.039037 5316 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:01.046811 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:11:01.048844 5314 leader_election.cc:341] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.052421 5399 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:01.053793 5399 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:01.055239 5399 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.061180 5450 raft_consensus.cc:2781] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:01.068261 5450 raft_consensus.cc:686] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
W20260929 14:11:01.077603 5239 consensus_peers.cc:487] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 -> Peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Couldn't send request to peer d886fb1eb0474d2d9f7410c4013c45e4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:01.080236 5450 consensus_queue.cc:237] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.100272 5392 tablet_bootstrap.cc:654] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.102206 5391 ts_tablet_manager.cc:1428] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.134s user 0.071s sys 0.061s
W20260929 14:11:01.102608 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 63666.3us
I20260929 14:11:01.103123 5391 tablet_bootstrap.cc:492] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:01.105181 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:01.110435 5391 tablet_bootstrap.cc:654] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.111711 5392 tablet_bootstrap.cc:492] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:01.112159 5392 ts_tablet_manager.cc:1397] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.080s user 0.011s sys 0.013s
I20260929 14:11:01.113366 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:01.113297 5097 catalog_manager.cc:5495] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:01.115260 5399 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:11:01.110322 5316 leader_election.cc:341] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
W20260929 14:11:01.116009 5314 leader_election.cc:341] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.118304 5314 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:01.115047 5392 raft_consensus.cc:348] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.119222 5392 raft_consensus.cc:374] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.119626 5392 raft_consensus.cc:729] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.119743 5399 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:01.120363 5392 consensus_queue.cc:260] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.122002 5391 tablet_bootstrap.cc:492] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:01.122466 5391 ts_tablet_manager.cc:1397] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.007s
I20260929 14:11:01.122701 5392 ts_tablet_manager.cc:1428] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.010s user 0.001s sys 0.007s
I20260929 14:11:01.123526 5392 tablet_bootstrap.cc:492] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:01.124816 5391 raft_consensus.cc:348] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.125767 5391 raft_consensus.cc:374] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.126025 5391 raft_consensus.cc:729] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.127038 5391 consensus_queue.cc:260] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.129184 5391 ts_tablet_manager.cc:1428] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:11:01.130060 5391 tablet_bootstrap.cc:492] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:01.130153 5392 tablet_bootstrap.cc:654] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.134075 5392 tablet_bootstrap.cc:492] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:01.134421 5392 ts_tablet_manager.cc:1397] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260929 14:11:01.136222 5391 tablet_bootstrap.cc:654] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.136788 5392 raft_consensus.cc:348] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.137629 5392 raft_consensus.cc:374] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.137907 5392 raft_consensus.cc:729] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.138767 5392 consensus_queue.cc:260] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.143013 5391 tablet_bootstrap.cc:492] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:01.143365 5392 ts_tablet_manager.cc:1428] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.009s user 0.003s sys 0.003s
I20260929 14:11:01.145454 5392 tablet_bootstrap.cc:492] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:01.143520 5391 ts_tablet_manager.cc:1397] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20260929 14:11:01.151863 5392 tablet_bootstrap.cc:654] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.152837 5391 raft_consensus.cc:348] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.153820 5391 raft_consensus.cc:374] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.154163 5391 raft_consensus.cc:729] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.155141 5391 consensus_queue.cc:260] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.158079 5391 ts_tablet_manager.cc:1428] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260929 14:11:01.159114 5393 ts_tablet_manager.cc:1428] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.192s user 0.010s sys 0.001s
I20260929 14:11:01.159663 5391 tablet_bootstrap.cc:492] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:01.160033 5393 tablet_bootstrap.cc:492] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:01.168630 5391 tablet_bootstrap.cc:654] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.182143 5398 consensus_queue.cc:1035] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:01.212033 5419 consensus_queue.cc:1035] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:01.215730 5450 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:01.216287 5450 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.218351 5450 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:01.218446 5099 catalog_manager.cc:5495] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:01.224200 5392 tablet_bootstrap.cc:492] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:01.224864 5392 ts_tablet_manager.cc:1397] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.080s user 0.021s sys 0.023s
I20260929 14:11:01.227197 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:01.228771 5392 raft_consensus.cc:348] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.229859 5392 raft_consensus.cc:374] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.230268 5392 raft_consensus.cc:729] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.231154 5392 consensus_queue.cc:260] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:01.232122 5316 leader_election.cc:341] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.233935 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:01.234326 5392 ts_tablet_manager.cc:1428] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260929 14:11:01.235620 5392 tablet_bootstrap.cc:492] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
W20260929 14:11:01.235900 5314 leader_election.cc:341] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.236328 5314 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:01.237674 5450 raft_consensus.cc:2726] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:01.239713 5200 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.151731 5275 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:01.249522 5419 consensus_queue.cc:1035] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:01.251187 5392 tablet_bootstrap.cc:654] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.259744 5450 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:01.260438 5450 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.265795 5450 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:01.267431 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:01.268746 5314 leader_election.cc:341] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.284493 5419 consensus_queue.cc:1035] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:01.291324 5392 tablet_bootstrap.cc:492] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:01.292032 5392 ts_tablet_manager.cc:1397] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.057s user 0.036s sys 0.012s
I20260929 14:11:01.295370 5392 raft_consensus.cc:348] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.296439 5392 raft_consensus.cc:374] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.296829 5392 raft_consensus.cc:729] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.297869 5392 consensus_queue.cc:260] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.300848 5392 ts_tablet_manager.cc:1428] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260929 14:11:01.302304 5392 tablet_bootstrap.cc:492] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:01.309764 5465 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:01.310557 5465 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.314492 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:01.315548 5201 raft_consensus.cc:2454] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:01.317135 5242 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:01.318719 5437 raft_consensus.cc:2781] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:01.319218 5437 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:01.319674 5437 raft_consensus.cc:3037] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.329118 5465 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:01.309335 5392 tablet_bootstrap.cc:654] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.331993 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:01.333621 5239 leader_election.cc:341] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.336447 5437 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.341135 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:01.342283 5201 raft_consensus.cc:3037] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.355772 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:11:01.357731 5239 leader_election.cc:341] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.359858 5465 consensus_queue.cc:1035] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:01.361461 5465 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:01.361882 5465 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.369446 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:01.370656 5198 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:01.370776 5465 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:01.374970 5242 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:01.375103 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:01.376428 5316 leader_election.cc:341] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.376869 5316 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:01.377630 5450 raft_consensus.cc:2726] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:01.359056 5437 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:01.381696 5465 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:01.382115 5465 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:01.382506 5465 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:11:01.385015 5239 consensus_peers.cc:487] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 -> Peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Couldn't send request to peer d886fb1eb0474d2d9f7410c4013c45e4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:01.352901 5201 raft_consensus.cc:2454] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:01.386942 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:01.388662 5242 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:01.389688 5418 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.390424 5419 raft_consensus.cc:2781] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
W20260929 14:11:01.390544 5239 leader_election.cc:341] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.391027 5419 raft_consensus.cc:686] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:01.392756 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:01.392650 5419 consensus_queue.cc:237] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.393445 5349 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.397469 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:01.398146 5200 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.400719 5349 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:01.401849 5239 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:01.407210 5447 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:01.413870 5437 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:01.414740 5437 raft_consensus.cc:686] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:01.416052 5437 consensus_queue.cc:237] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.417294 5393 tablet_bootstrap.cc:700] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent opening tablet: real 0.100s user 0.004s sys 0.000s
I20260929 14:11:01.423801 5465 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.480785 5393 tablet_bootstrap.cc:654] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.482084 5465 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:01.451050 5447 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:01.452437 5200 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:01.452306 5473 consensus_queue.cc:1035] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:01.482858 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:01.484979 5199 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.488776 5418 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:01.488224 5393 tablet_bootstrap.cc:492] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:01.489892 5393 ts_tablet_manager.cc:1397] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.330s user 0.017s sys 0.000s
I20260929 14:11:01.491611 5393 raft_consensus.cc:348] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.492136 5393 raft_consensus.cc:374] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.492288 5393 raft_consensus.cc:729] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.492815 5393 consensus_queue.cc:260] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.495215 5393 ts_tablet_manager.cc:1428] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:11:01.495386 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:01.496006 5393 tablet_bootstrap.cc:492] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:01.496939 5275 raft_consensus.cc:3037] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.501647 5393 tablet_bootstrap.cc:654] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
W20260929 14:11:01.504384 5239 leader_election.cc:341] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.504908 5391 tablet_bootstrap.cc:492] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:01.505350 5391 ts_tablet_manager.cc:1397] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.346s user 0.064s sys 0.109s
I20260929 14:11:01.506577 5393 tablet_bootstrap.cc:492] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:01.506958 5393 ts_tablet_manager.cc:1397] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260929 14:11:01.508224 5391 raft_consensus.cc:348] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.508955 5391 raft_consensus.cc:374] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.510077 5450 consensus_queue.cc:1035] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:01.517895 5391 raft_consensus.cc:729] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.518659 5450 consensus_queue.cc:1035] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:01.518610 5391 consensus_queue.cc:260] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.520670 5391 ts_tablet_manager.cc:1428] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20260929 14:11:01.521943 5391 tablet_bootstrap.cc:492] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:01.512670 5393 raft_consensus.cc:348] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.513579 5392 tablet_bootstrap.cc:492] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:01.523247 5393 raft_consensus.cc:374] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.523531 5393 raft_consensus.cc:729] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.523643 5392 ts_tablet_manager.cc:1397] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.222s user 0.022s sys 0.000s
I20260929 14:11:01.524492 5393 consensus_queue.cc:260] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.527045 5392 raft_consensus.cc:348] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.517011 5397 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:01.538487 5397 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.540628 5397 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:01.541479 5391 tablet_bootstrap.cc:654] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.543838 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:01.544461 5276 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:01.545794 5165 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:01.546692 5397 raft_consensus.cc:2781] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:01.547036 5397 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:01.547349 5397 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.549679 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:01.550985 5164 leader_election.cc:341] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.516199 5199 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:01.528064 5393 ts_tablet_manager.cc:1428] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.021s user 0.007s sys 0.000s
I20260929 14:11:01.555593 5393 tablet_bootstrap.cc:492] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:01.529343 5419 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:01.557621 5419 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.559093 5242 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:01.560137 5419 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:01.562642 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:01.562783 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:01.564491 5242 leader_election.cc:341] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.553882 5392 raft_consensus.cc:374] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.566213 5392 raft_consensus.cc:729] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.567351 5392 consensus_queue.cc:260] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.560802 5419 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:01.582118 5419 raft_consensus.cc:686] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:01.583190 5418 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:01.583794 5418 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.586978 5418 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:01.590894 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:01.592871 5242 leader_election.cc:341] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.573302 5392 ts_tablet_manager.cc:1428] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.049s user 0.008s sys 0.000s
W20260929 14:11:01.578012 5239 leader_election.cc:341] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.595048 5392 tablet_bootstrap.cc:492] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:01.599354 5239 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:01.602190 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:01.602587 5147 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260929 14:11:01.602900 5351 raft_consensus.cc:2454] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:01.580574 5397 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.614527 5418 raft_consensus.cc:2726] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:01.615931 5239 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:01.588762 5419 consensus_queue.cc:237] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.625552 5474 raft_consensus.cc:2781] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:01.631219 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:01.654621 5276 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.629861 5474 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:01.644888 5397 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:01.695832 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 85484.7us
I20260929 14:11:01.696650 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:11:01.698169 5164 leader_election.cc:341] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.629938 5098 catalog_manager.cc:5495] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:01.687388 5393 tablet_bootstrap.cc:654] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.699339 5474 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.708751 5391 tablet_bootstrap.cc:492] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:01.709208 5391 ts_tablet_manager.cc:1397] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.187s user 0.025s sys 0.045s
I20260929 14:11:01.711040 5098 catalog_manager.cc:5495] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:01.712095 5391 raft_consensus.cc:348] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.712805 5391 raft_consensus.cc:374] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.713078 5391 raft_consensus.cc:729] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.713618 5276 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:01.713836 5391 consensus_queue.cc:260] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.714834 5165 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:01.716804 5418 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:01.718155 5445 raft_consensus.cc:2781] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:01.719238 5392 tablet_bootstrap.cc:654] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.719470 5391 ts_tablet_manager.cc:1428] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260929 14:11:01.718240 5418 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.718899 5445 raft_consensus.cc:686] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:01.720701 5391 tablet_bootstrap.cc:492] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:01.720600 5445 consensus_queue.cc:237] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.721268 5418 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:01.722743 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:01.723573 5351 raft_consensus.cc:2454] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:01.725835 5239 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:01.726290 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:01.726809 5418 raft_consensus.cc:2781] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:01.727248 5418 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:01.727576 5418 raft_consensus.cc:3037] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:11:01.728132 5242 leader_election.cc:341] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.731706 5419 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:01.732146 5419 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.730021 5474 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.739740 5419 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:01.740164 5474 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:01.741984 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:01.743144 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:01.743147 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:11:01.745786 5239 leader_election.cc:341] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.746237 5349 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:11:01.746675 5242 leader_election.cc:341] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.747076 5242 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:01.748703 5419 raft_consensus.cc:2726] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:01.749544 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:11:01.750792 5242 leader_election.cc:341] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.766799 5349 raft_consensus.cc:2454] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:01.770272 5239 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:01.771371 5474 raft_consensus.cc:2781] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:01.771693 5391 tablet_bootstrap.cc:654] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.771584 5393 tablet_bootstrap.cc:492] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:01.771814 5474 raft_consensus.cc:686] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:01.772770 5393 ts_tablet_manager.cc:1397] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.217s user 0.021s sys 0.028s
I20260929 14:11:01.773109 5474 consensus_queue.cc:237] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.775931 5392 tablet_bootstrap.cc:492] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:01.776383 5392 ts_tablet_manager.cc:1397] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.182s user 0.008s sys 0.004s
I20260929 14:11:01.776692 5393 raft_consensus.cc:348] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.777804 5393 raft_consensus.cc:374] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.778134 5393 raft_consensus.cc:729] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.779239 5393 consensus_queue.cc:260] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.780043 5392 raft_consensus.cc:348] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.780885 5392 raft_consensus.cc:374] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.781147 5392 raft_consensus.cc:729] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.762126 5418 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.784199 5419 consensus_queue.cc:1035] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 14:11:01.784875 5239 consensus_peers.cc:487] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 -> Peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Couldn't send request to peer d886fb1eb0474d2d9f7410c4013c45e4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:01.785274 5418 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:01.786638 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:01.787257 5393 ts_tablet_manager.cc:1428] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.014s user 0.003s sys 0.005s
I20260929 14:11:01.787202 5349 raft_consensus.cc:3037] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.788374 5393 tablet_bootstrap.cc:492] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:01.789031 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:11:01.791200 5242 leader_election.cc:341] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.796039 5393 tablet_bootstrap.cc:654] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.801821 5392 consensus_queue.cc:260] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.807405 5392 ts_tablet_manager.cc:1428] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20260929 14:11:01.808449 5392 tablet_bootstrap.cc:492] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:01.811721 5393 tablet_bootstrap.cc:492] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:01.812163 5393 ts_tablet_manager.cc:1397] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.004s
I20260929 14:11:01.813772 5474 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:01.814240 5393 raft_consensus.cc:348] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.814903 5393 raft_consensus.cc:374] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.814661 5474 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.815173 5393 raft_consensus.cc:729] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.815788 5393 consensus_queue.cc:260] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.816643 5392 tablet_bootstrap.cc:654] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.816958 5474 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:01.817963 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:01.818445 5349 raft_consensus.cc:2454] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:01.818612 5351 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:01.825958 5239 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:01.826777 5418 raft_consensus.cc:2781] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:01.827073 5239 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:01.827258 5418 raft_consensus.cc:686] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:01.828025 5418 consensus_queue.cc:237] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.830196 5437 raft_consensus.cc:2781] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:01.830562 5437 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:01.830880 5437 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.836575 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:01.837677 5201 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:01.838779 5097 catalog_manager.cc:5495] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:01.842800 5437 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.843212 5097 catalog_manager.cc:5495] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:01.844262 5393 ts_tablet_manager.cc:1428] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20260929 14:11:01.849107 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:01.849911 5201 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.852046 5393 tablet_bootstrap.cc:492] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:01.856379 5391 tablet_bootstrap.cc:492] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:01.856918 5391 ts_tablet_manager.cc:1397] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.136s user 0.013s sys 0.020s
I20260929 14:11:01.859772 5399 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:01.860356 5399 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.863739 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:01.864501 5351 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:01.864936 5098 catalog_manager.cc:5495] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:01.866719 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:01.868453 5316 leader_election.cc:341] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.870322 5399 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:01.870476 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:01.874078 5314 leader_election.cc:341] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.874640 5314 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:01.876134 5399 raft_consensus.cc:2726] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:01.876365 5393 tablet_bootstrap.cc:654] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.887734 5351 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:01.889853 5239 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:01.890633 5418 raft_consensus.cc:2781] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:01.891485 5418 raft_consensus.cc:686] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:01.892624 5418 consensus_queue.cc:237] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.894814 5437 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:01.897114 5099 catalog_manager.cc:5495] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:01.901815 5391 raft_consensus.cc:348] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.902752 5391 raft_consensus.cc:374] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.903091 5391 raft_consensus.cc:729] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.907274 5201 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:01.917312 5437 consensus_queue.cc:1035] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:01.926563 5399 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:01.927199 5399 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.931450 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:01.932067 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:01.932808 5314 leader_election.cc:341] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.933192 5399 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:11:01.934648 5316 leader_election.cc:341] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:01.935220 5316 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:01.936537 5399 raft_consensus.cc:2726] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:01.937492 5393 tablet_bootstrap.cc:492] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:01.937942 5393 ts_tablet_manager.cc:1397] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.086s user 0.012s sys 0.000s
I20260929 14:11:01.940887 5393 raft_consensus.cc:348] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.941669 5393 raft_consensus.cc:374] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.941954 5393 raft_consensus.cc:729] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.942654 5393 consensus_queue.cc:260] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.943881 5474 consensus_queue.cc:1035] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:01.946224 5097 catalog_manager.cc:5495] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:01.912005 5391 consensus_queue.cc:260] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.965978 5392 tablet_bootstrap.cc:492] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:01.966437 5392 ts_tablet_manager.cc:1397] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.158s user 0.032s sys 0.073s
I20260929 14:11:01.967358 5391 ts_tablet_manager.cc:1428] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.069s user 0.004s sys 0.003s
I20260929 14:11:01.969156 5392 raft_consensus.cc:348] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.969866 5392 raft_consensus.cc:374] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:01.970108 5392 raft_consensus.cc:729] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:01.970683 5392 consensus_queue.cc:260] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:01.972606 5392 ts_tablet_manager.cc:1428] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:11:01.973371 5392 tablet_bootstrap.cc:492] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:01.975626 5391 tablet_bootstrap.cc:492] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:01.980528 5392 tablet_bootstrap.cc:654] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:01.983798 5393 ts_tablet_manager.cc:1428] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.046s user 0.008s sys 0.000s
I20260929 14:11:01.984707 5393 tablet_bootstrap.cc:492] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:01.993923 5391 tablet_bootstrap.cc:654] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.000015 5393 tablet_bootstrap.cc:654] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.010666 5445 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:02.011193 5445 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.013581 5445 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.014138 5391 tablet_bootstrap.cc:492] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:02.014102 5393 tablet_bootstrap.cc:492] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:02.014748 5391 ts_tablet_manager.cc:1397] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.001s
I20260929 14:11:02.014829 5393 ts_tablet_manager.cc:1397] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.002s
I20260929 14:11:02.015838 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:02.016451 5276 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:02.017362 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:02.017294 5391 raft_consensus.cc:348] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.017591 5393 raft_consensus.cc:348] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.018023 5349 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:02.018112 5391 raft_consensus.cc:374] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.018584 5391 raft_consensus.cc:729] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.018913 5165 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:02.019157 5393 raft_consensus.cc:374] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.019263 5391 consensus_queue.cc:260] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.019531 5393 raft_consensus.cc:729] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.020309 5447 raft_consensus.cc:2781] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:02.020686 5447 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:02.021030 5447 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.021448 5399 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:02.021848 5399 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.023869 5399 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:02.025889 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:02.026098 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:02.027212 5316 leader_election.cc:341] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
W20260929 14:11:02.027679 5314 leader_election.cc:341] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.028093 5314 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:02.029126 5399 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:02.020834 5393 consensus_queue.cc:260] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.030987 5447 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.031525 5393 ts_tablet_manager.cc:1428] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260929 14:11:02.032415 5393 tablet_bootstrap.cc:492] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:02.033110 5447 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.034845 5391 ts_tablet_manager.cc:1428] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.020s user 0.003s sys 0.004s
I20260929 14:11:02.034715 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:02.035274 5349 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.035748 5391 tablet_bootstrap.cc:492] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:02.038362 5393 tablet_bootstrap.cc:654] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.029902 5392 tablet_bootstrap.cc:492] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:02.040072 5392 ts_tablet_manager.cc:1397] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.067s user 0.014s sys 0.013s
I20260929 14:11:02.041994 5391 tablet_bootstrap.cc:654] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.043040 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:02.043689 5276 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.043623 5392 raft_consensus.cc:348] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.049623 5392 raft_consensus.cc:374] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.049930 5392 raft_consensus.cc:729] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.055088 5349 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:02.055752 5392 consensus_queue.cc:260] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.059221 5276 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:02.082360 5164 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:02.087098 5447 raft_consensus.cc:2781] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:02.087641 5447 raft_consensus.cc:686] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:02.088440 5447 consensus_queue.cc:237] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.091410 5392 ts_tablet_manager.cc:1428] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.051s user 0.015s sys 0.000s
I20260929 14:11:02.097553 5392 tablet_bootstrap.cc:492] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:02.102818 5349 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.113153 5437 consensus_queue.cc:1035] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:02.120352 5391 tablet_bootstrap.cc:492] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:02.120803 5391 ts_tablet_manager.cc:1397] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.085s user 0.019s sys 0.043s
I20260929 14:11:02.122319 5393 tablet_bootstrap.cc:492] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:02.122754 5393 ts_tablet_manager.cc:1397] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.090s user 0.022s sys 0.006s
I20260929 14:11:02.129469 5392 tablet_bootstrap.cc:654] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.130766 5391 raft_consensus.cc:348] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.131465 5391 raft_consensus.cc:374] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.131724 5391 raft_consensus.cc:729] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.132539 5391 consensus_queue.cc:260] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.132958 5098 catalog_manager.cc:5495] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:02.134817 5391 ts_tablet_manager.cc:1428] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.014s user 0.004s sys 0.004s
I20260929 14:11:02.135681 5391 tablet_bootstrap.cc:492] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:02.140362 5393 raft_consensus.cc:348] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.141026 5393 raft_consensus.cc:374] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.141287 5393 raft_consensus.cc:729] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.142056 5393 consensus_queue.cc:260] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.143785 5393 ts_tablet_manager.cc:1428] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260929 14:11:02.144454 5393 tablet_bootstrap.cc:492] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:02.148682 5391 tablet_bootstrap.cc:654] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.159016 5391 tablet_bootstrap.cc:492] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:02.159561 5391 ts_tablet_manager.cc:1397] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.002s
I20260929 14:11:02.162019 5391 raft_consensus.cc:348] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.162878 5391 raft_consensus.cc:374] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.163143 5391 raft_consensus.cc:729] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.163858 5391 consensus_queue.cc:260] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.166234 5391 ts_tablet_manager.cc:1428] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:11:02.167130 5391 tablet_bootstrap.cc:492] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:02.149816 5393 tablet_bootstrap.cc:654] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.151333 5465 consensus_queue.cc:1035] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:02.173233 5391 tablet_bootstrap.cc:654] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.191174 5393 tablet_bootstrap.cc:492] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:02.191646 5393 ts_tablet_manager.cc:1397] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.003s
I20260929 14:11:02.193994 5393 raft_consensus.cc:348] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.194689 5393 raft_consensus.cc:374] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.194960 5393 raft_consensus.cc:729] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.196280 5393 consensus_queue.cc:260] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.197438 5392 tablet_bootstrap.cc:492] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:02.197934 5392 ts_tablet_manager.cc:1397] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.104s user 0.015s sys 0.006s
I20260929 14:11:02.199662 5465 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:02.200301 5465 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.204120 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:02.200968 5392 raft_consensus.cc:348] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.205000 5349 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:02.209887 5447 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:02.210450 5447 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.211189 5239 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:02.212342 5471 raft_consensus.cc:2781] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:02.212564 5447 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.212847 5471 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:02.213581 5471 raft_consensus.cc:3037] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.216260 5393 ts_tablet_manager.cc:1428] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.024s user 0.008s sys 0.001s
I20260929 14:11:02.217324 5393 tablet_bootstrap.cc:492] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:02.217940 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:02.218793 5349 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:02.220178 5164 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:02.205550 5465 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.223685 5393 tablet_bootstrap.cc:654] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.206214 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:02.225664 5198 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:02.228595 5471 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.232564 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:02.233363 5349 raft_consensus.cc:3037] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.234146 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:02.234782 5198 raft_consensus.cc:3037] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.235936 5393 tablet_bootstrap.cc:492] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:02.236459 5393 ts_tablet_manager.cc:1397] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.007s
I20260929 14:11:02.220662 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:02.237911 5276 raft_consensus.cc:2379] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 1: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:02.239133 5471 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.239782 5393 raft_consensus.cc:348] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.240760 5393 raft_consensus.cc:374] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.241082 5393 raft_consensus.cc:729] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.242039 5393 consensus_queue.cc:260] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.242905 5349 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:02.244355 5239 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:02.244717 5393 ts_tablet_manager.cc:1428] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.008s user 0.000s sys 0.006s
I20260929 14:11:02.245491 5471 raft_consensus.cc:2781] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:02.246562 5471 raft_consensus.cc:686] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:02.244841 5198 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:02.247283 5447 raft_consensus.cc:2741] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260929 14:11:02.247340 5471 consensus_queue.cc:237] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.245831 5392 raft_consensus.cc:374] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.249768 5392 raft_consensus.cc:729] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.250022 5391 tablet_bootstrap.cc:492] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:02.250474 5391 ts_tablet_manager.cc:1397] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.084s user 0.045s sys 0.035s
I20260929 14:11:02.250530 5392 consensus_queue.cc:260] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.245730 5393 tablet_bootstrap.cc:492] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:02.253449 5391 raft_consensus.cc:348] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.254172 5391 raft_consensus.cc:374] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.254462 5391 raft_consensus.cc:729] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.255060 5391 consensus_queue.cc:260] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.256867 5391 ts_tablet_manager.cc:1428] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:11:02.253398 5392 ts_tablet_manager.cc:1428] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.055s user 0.007s sys 0.000s
I20260929 14:11:02.257807 5391 tablet_bootstrap.cc:492] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:02.258126 5392 tablet_bootstrap.cc:492] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:02.259057 5393 tablet_bootstrap.cc:654] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.259519 5097 catalog_manager.cc:5495] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:02.264635 5391 tablet_bootstrap.cc:654] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.266842 5392 tablet_bootstrap.cc:654] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.271653 5447 consensus_queue.cc:1035] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:02.274533 5399 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:02.275023 5399 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.277096 5399 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:02.279531 5447 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:02.280078 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:02.280014 5447 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.276487 5391 tablet_bootstrap.cc:492] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:02.281584 5391 ts_tablet_manager.cc:1397] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.024s user 0.010s sys 0.008s
W20260929 14:11:02.281898 5316 leader_election.cc:341] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.283835 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:02.284112 5391 raft_consensus.cc:348] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.284754 5391 raft_consensus.cc:374] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.285066 5391 raft_consensus.cc:729] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.281590 5471 consensus_queue.cc:1035] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:02.285382 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:02.285655 5391 consensus_queue.cc:260] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.286723 5349 raft_consensus.cc:2379] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 1: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
W20260929 14:11:02.285408 5314 leader_election.cc:341] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.287344 5314 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:02.288291 5391 ts_tablet_manager.cc:1428] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:11:02.289597 5450 raft_consensus.cc:2726] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:02.289759 5391 tablet_bootstrap.cc:492] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:02.292091 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:02.294099 5471 consensus_queue.cc:1035] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:02.296130 5391 tablet_bootstrap.cc:654] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.297290 5165 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:02.298463 5397 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.303526 5392 tablet_bootstrap.cc:492] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
W20260929 14:11:02.305773 5164 consensus_peers.cc:487] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd -> Peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Couldn't send request to peer d886fb1eb0474d2d9f7410c4013c45e4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:02.306663 5392 ts_tablet_manager.cc:1397] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.049s user 0.014s sys 0.004s
I20260929 14:11:02.310305 5445 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:02.311156 5397 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:02.310835 5445 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.308048 5447 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.312911 5445 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.318058 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:02.321115 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:02.321997 5276 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:02.319047 5349 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:02.323781 5164 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:02.325779 5445 raft_consensus.cc:2781] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:02.326159 5445 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:02.326502 5445 raft_consensus.cc:3037] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.333460 5419 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:02.333904 5465 consensus_queue.cc:1035] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:02.333966 5419 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.336046 5419 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.337457 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:02.338079 5350 raft_consensus.cc:2454] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
W20260929 14:11:02.338788 5242 consensus_peers.cc:487] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 -> Peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Couldn't send request to peer ac17d3b553b74c9da39e0ab307addacd. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:02.339200 5239 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:02.341246 5418 raft_consensus.cc:2781] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:02.341596 5418 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:02.341873 5418 raft_consensus.cc:3037] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.347100 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:02.347811 5198 raft_consensus.cc:2454] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:02.347699 5419 consensus_queue.cc:1035] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:02.348779 5418 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.352865 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:02.353828 5199 raft_consensus.cc:3037] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.359611 5393 tablet_bootstrap.cc:492] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:02.360093 5393 ts_tablet_manager.cc:1397] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.115s user 0.008s sys 0.016s
I20260929 14:11:02.363212 5391 tablet_bootstrap.cc:492] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:02.363667 5391 ts_tablet_manager.cc:1397] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.074s user 0.018s sys 0.021s
I20260929 14:11:02.363541 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:02.364111 5349 raft_consensus.cc:3037] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.370600 5391 raft_consensus.cc:348] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.372845 5392 raft_consensus.cc:348] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.373922 5391 raft_consensus.cc:374] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.374083 5392 raft_consensus.cc:374] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.374092 5450 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:02.374432 5391 raft_consensus.cc:729] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.374748 5450 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.375229 5391 consensus_queue.cc:260] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.376616 5450 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:02.377697 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:02.378176 5391 ts_tablet_manager.cc:1428] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.014s user 0.003s sys 0.003s
I20260929 14:11:02.373431 5418 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.379522 5391 tablet_bootstrap.cc:492] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:02.372437 5349 raft_consensus.cc:2454] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:02.379426 5199 raft_consensus.cc:2454] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:02.379604 5445 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.380671 5239 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:02.382457 5419 raft_consensus.cc:2781] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:02.382936 5419 raft_consensus.cc:686] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:02.383111 5392 raft_consensus.cc:729] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.383701 5419 consensus_queue.cc:237] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.383757 5392 consensus_queue.cc:260] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.393234 5445 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.394531 5392 ts_tablet_manager.cc:1428] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.087s user 0.005s sys 0.001s
I20260929 14:11:02.395390 5392 tablet_bootstrap.cc:492] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:02.396763 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:02.397429 5276 raft_consensus.cc:3037] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.398536 5465 consensus_queue.cc:1035] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:02.400583 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:02.401144 5349 raft_consensus.cc:3037] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.391067 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:02.364114 5393 raft_consensus.cc:348] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } }
I20260929 14:11:02.407349 5393 raft_consensus.cc:374] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.407609 5393 raft_consensus.cc:729] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.408433 5316 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:02.409394 5450 raft_consensus.cc:2726] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:02.410370 5393 consensus_queue.cc:260] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } }
I20260929 14:11:02.411072 5391 tablet_bootstrap.cc:654] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.412580 5418 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:02.412709 5393 ts_tablet_manager.cc:1428] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.052s user 0.005s sys 0.001s
I20260929 14:11:02.413113 5418 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.413820 5393 tablet_bootstrap.cc:492] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:02.416399 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:02.416749 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:02.416976 5350 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
W20260929 14:11:02.422214 5222 debug-util.cc:398] Leaking SignalData structure 0x7b08000e36e0 after lost signal to thread 4044
I20260929 14:11:02.415648 5418 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.440650 5447 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:02.452869 5276 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
W20260929 14:11:02.447041 5242 leader_election.cc:341] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.463634 5165 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:02.466346 5397 raft_consensus.cc:2781] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:02.466965 5397 raft_consensus.cc:686] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:02.468094 5397 consensus_queue.cc:237] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.447667 5447 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.479877 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:02.480618 5276 raft_consensus.cc:2454] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:02.481693 5447 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.485441 5165 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:02.486362 5447 raft_consensus.cc:2781] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:02.486275 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:02.486850 5447 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 14:11:02.488746 5164 leader_election.cc:341] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.474838 5239 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:02.487282 5447 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.479089 5349 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:02.500471 5392 tablet_bootstrap.cc:654] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
W20260929 14:11:02.501570 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 55131.6us
I20260929 14:11:02.502326 5419 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:02.502698 5419 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:02.503028 5419 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.503217 5099 catalog_manager.cc:5495] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:02.505254 5349 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:11:02.496299 5222 debug-util.cc:398] Leaking SignalData structure 0x7b0800083d80 after lost signal to thread 5133
I20260929 14:11:02.509626 5098 catalog_manager.cc:5495] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:02.516841 5447 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.531345 5447 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.529835 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:02.532713 5276 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.532594 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:11:02.534029 5164 leader_election.cc:341] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
W20260929 14:11:02.529320 5222 debug-util.cc:398] Leaking SignalData structure 0x7b0800057340 after lost signal to thread 5225
I20260929 14:11:02.537364 5419 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.539597 5419 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.540282 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:02.540911 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:02.541566 5351 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:11:02.541563 5242 leader_election.cc:341] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.543471 5393 tablet_bootstrap.cc:654] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
W20260929 14:11:02.540958 5222 debug-util.cc:398] Leaking SignalData structure 0x7b08000bfe60 after lost signal to thread 5300
I20260929 14:11:02.547930 5447 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:02.548633 5222 debug-util.cc:398] Leaking SignalData structure 0x7b08001bfee0 after lost signal to thread 5375
I20260929 14:11:02.548368 5447 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.551143 5350 raft_consensus.cc:3037] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.555095 5391 tablet_bootstrap.cc:492] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:02.555227 5276 raft_consensus.cc:2454] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:02.555619 5391 ts_tablet_manager.cc:1397] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.177s user 0.034s sys 0.017s
I20260929 14:11:02.556452 5165 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:02.557595 5397 raft_consensus.cc:2781] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:02.558032 5397 raft_consensus.cc:686] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:02.558775 5397 consensus_queue.cc:237] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.561344 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:02.562732 5165 leader_election.cc:341] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.563035 5391 raft_consensus.cc:348] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.564894 5391 raft_consensus.cc:374] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.565932 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:02.566591 5348 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:02.567898 5164 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:02.568938 5447 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.569124 5445 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:02.565169 5391 raft_consensus.cc:729] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.570698 5391 consensus_queue.cc:260] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.569612 5445 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:02.574333 5445 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.574601 5391 ts_tablet_manager.cc:1428] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260929 14:11:02.575026 5392 tablet_bootstrap.cc:492] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:02.575527 5391 tablet_bootstrap.cc:492] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:02.575701 5392 ts_tablet_manager.cc:1397] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.180s user 0.021s sys 0.001s
I20260929 14:11:02.581475 5392 raft_consensus.cc:348] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.584789 5392 raft_consensus.cc:374] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.585776 5392 raft_consensus.cc:729] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.587685 5392 consensus_queue.cc:260] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.591691 5447 consensus_queue.cc:1035] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:02.594880 5351 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:02.602632 5392 ts_tablet_manager.cc:1428] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.026s user 0.008s sys 0.000s
I20260929 14:11:02.604959 5239 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:02.606437 5391 tablet_bootstrap.cc:654] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.606983 5392 tablet_bootstrap.cc:492] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:02.612152 5419 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:02.614409 5419 raft_consensus.cc:686] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:02.615206 5419 consensus_queue.cc:237] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.605979 5445 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.630565 5445 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.631093 5445 consensus_queue.cc:1035] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:02.632382 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:02.632951 5351 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.632853 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:02.635535 5097 catalog_manager.cc:5495] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
W20260929 14:11:02.641422 5165 leader_election.cc:341] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.650447 5392 tablet_bootstrap.cc:654] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.650588 5393 tablet_bootstrap.cc:492] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:02.650978 5393 ts_tablet_manager.cc:1397] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.237s user 0.009s sys 0.010s
I20260929 14:11:02.651801 5474 consensus_queue.cc:1035] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:02.689575 5393 raft_consensus.cc:348] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.690970 5393 raft_consensus.cc:374] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.691498 5393 raft_consensus.cc:729] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.692389 5393 consensus_queue.cc:260] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.701166 5393 ts_tablet_manager.cc:1428] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.050s user 0.008s sys 0.004s
I20260929 14:11:02.702109 5393 tablet_bootstrap.cc:492] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:02.704633 5392 tablet_bootstrap.cc:492] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:02.705083 5392 ts_tablet_manager.cc:1397] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.098s user 0.020s sys 0.018s
I20260929 14:11:02.712178 5098 catalog_manager.cc:5495] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:02.714445 5391 tablet_bootstrap.cc:492] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:02.714984 5391 ts_tablet_manager.cc:1397] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.140s user 0.014s sys 0.009s
I20260929 14:11:02.718777 5351 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
W20260929 14:11:02.721306 5147 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260929 14:11:02.721189 5392 raft_consensus.cc:348] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.721331 5450 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:02.722250 5392 raft_consensus.cc:374] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.722628 5392 raft_consensus.cc:729] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.722332 5450 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.723348 5392 consensus_queue.cc:260] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.724180 5393 tablet_bootstrap.cc:654] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.724278 5450 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:02.726684 5164 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:02.727497 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:02.727600 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:02.728231 5445 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:02.728668 5445 raft_consensus.cc:686] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
W20260929 14:11:02.728930 5316 leader_election.cc:341] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
W20260929 14:11:02.729890 5314 leader_election.cc:341] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.729542 5445 consensus_queue.cc:237] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.730304 5314 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:02.731886 5450 raft_consensus.cc:2726] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:02.732282 5392 ts_tablet_manager.cc:1428] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.027s user 0.006s sys 0.000s
I20260929 14:11:02.733273 5392 tablet_bootstrap.cc:492] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:02.734297 5391 raft_consensus.cc:348] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.734808 5391 raft_consensus.cc:374] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.735023 5391 raft_consensus.cc:729] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.735535 5391 consensus_queue.cc:260] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.745697 5392 tablet_bootstrap.cc:654] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.749909 5531 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:02.750581 5531 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.760298 5391 ts_tablet_manager.cc:1428] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.045s user 0.005s sys 0.002s
I20260929 14:11:02.762055 5391 tablet_bootstrap.cc:492] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:02.768134 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:02.768378 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:02.768929 5351 raft_consensus.cc:2454] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:02.770009 5164 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:02.770826 5397 raft_consensus.cc:2781] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:02.771183 5397 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:02.771068 5474 consensus_queue.cc:1035] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:02.771548 5397 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.774897 5531 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.779628 5531 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:02.775370 5165 leader_election.cc:341] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.780017 5531 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.784188 5393 tablet_bootstrap.cc:492] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:02.784626 5393 ts_tablet_manager.cc:1397] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.083s user 0.008s sys 0.010s
I20260929 14:11:02.787061 5531 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.788054 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:02.792079 5397 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.792424 5098 catalog_manager.cc:5495] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:02.796168 5474 consensus_queue.cc:1035] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:02.797993 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:02.805927 5393 raft_consensus.cc:348] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.806951 5393 raft_consensus.cc:374] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.807278 5393 raft_consensus.cc:729] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.808046 5393 consensus_queue.cc:260] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.815793 5391 tablet_bootstrap.cc:654] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.816938 5397 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.818352 5393 ts_tablet_manager.cc:1428] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.033s user 0.004s sys 0.004s
I20260929 14:11:02.819504 5392 tablet_bootstrap.cc:492] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:02.819693 5393 tablet_bootstrap.cc:492] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:02.818082 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:02.819921 5392 ts_tablet_manager.cc:1397] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.087s user 0.021s sys 0.005s
I20260929 14:11:02.822510 5392 raft_consensus.cc:348] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.826618 5392 raft_consensus.cc:374] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.826876 5392 raft_consensus.cc:729] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.827451 5392 consensus_queue.cc:260] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.820358 5349 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.834231 5392 ts_tablet_manager.cc:1428] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20260929 14:11:02.835011 5392 tablet_bootstrap.cc:492] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:02.830423 5165 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:02.836524 5397 raft_consensus.cc:2726] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:02.842010 5349 raft_consensus.cc:2454] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:02.842725 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:02.843180 5164 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:02.845453 5397 raft_consensus.cc:2781] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:02.845887 5397 raft_consensus.cc:686] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:02.846956 5397 consensus_queue.cc:237] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.849444 5418 consensus_queue.cc:1035] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:02.858955 5419 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:02.859371 5165 leader_election.cc:341] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.859580 5419 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.868498 5445 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:02.868949 5445 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.870128 5419 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.870837 5445 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.874476 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:02.875682 5164 leader_election.cc:341] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.876407 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:02.877795 5393 tablet_bootstrap.cc:654] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
W20260929 14:11:02.877827 5239 leader_election.cc:341] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.883075 5392 tablet_bootstrap.cc:654] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.885758 5391 tablet_bootstrap.cc:492] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:02.886179 5391 ts_tablet_manager.cc:1397] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.124s user 0.017s sys 0.027s
I20260929 14:11:02.887666 5199 raft_consensus.cc:3037] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.889434 5391 raft_consensus.cc:348] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.890395 5391 raft_consensus.cc:374] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:02.890734 5391 raft_consensus.cc:729] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.891844 5391 consensus_queue.cc:260] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.895272 5391 ts_tablet_manager.cc:1428] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260929 14:11:02.896916 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:02.897635 5201 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:02.898355 5445 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:02.899248 5445 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.901399 5445 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.904170 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:02.904826 5351 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:02.906450 5164 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:02.907296 5397 raft_consensus.cc:2781] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:02.907668 5397 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:02.908015 5397 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.914508 5531 consensus_queue.cc:1035] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:02.917599 5391 tablet_bootstrap.cc:492] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:02.923735 5465 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:02.924187 5465 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.928567 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:02.927064 5465 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.928362 5098 catalog_manager.cc:5495] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:02.929180 5276 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:02.929530 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:02.931069 5239 leader_election.cc:341] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.922248 5397 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.931571 5242 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:02.932727 5391 tablet_bootstrap.cc:654] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:02.932813 5465 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:02.933293 5465 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:02.933686 5465 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.934136 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:02.934710 5348 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.938580 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:02.942106 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:02.943774 5397 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:02.944008 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:02.944451 5348 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:02.945976 5164 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:02.947098 5397 raft_consensus.cc:2781] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:02.947645 5397 raft_consensus.cc:686] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:02.949484 5397 consensus_queue.cc:237] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:02.955521 5242 leader_election.cc:341] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.955955 5242 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:02.956693 5418 raft_consensus.cc:2726] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:02.958591 5165 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:02.965304 5165 leader_election.cc:341] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.965432 5445 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:02.965937 5445 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 14:11:02.966212 5165 leader_election.cc:341] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:02.966276 5445 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:02.970571 5392 tablet_bootstrap.cc:492] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:02.971012 5392 ts_tablet_manager.cc:1397] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.136s user 0.012s sys 0.018s
I20260929 14:11:02.971276 5391 tablet_bootstrap.cc:492] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:02.971740 5391 ts_tablet_manager.cc:1397] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.054s user 0.013s sys 0.003s
I20260929 14:11:02.974552 5391 raft_consensus.cc:348] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.975226 5391 raft_consensus.cc:374] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260929 14:11:02.975322 5164 consensus_peers.cc:487] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd -> Peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Couldn't send request to peer d886fb1eb0474d2d9f7410c4013c45e4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:02.975503 5391 raft_consensus.cc:729] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:02.976281 5465 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.976310 5391 consensus_queue.cc:260] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:02.988196 5393 tablet_bootstrap.cc:492] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:02.988684 5393 ts_tablet_manager.cc:1397] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.169s user 0.009s sys 0.003s
I20260929 14:11:02.991027 5393 raft_consensus.cc:348] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.006407 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:03.032219 5393 raft_consensus.cc:374] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.002446 5391 ts_tablet_manager.cc:1428] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.030s user 0.009s sys 0.000s
I20260929 14:11:03.033416 5391 tablet_bootstrap.cc:492] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:03.032583 5393 raft_consensus.cc:729] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.034327 5393 consensus_queue.cc:260] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.036468 5393 ts_tablet_manager.cc:1428] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.048s user 0.001s sys 0.004s
I20260929 14:11:03.007411 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:03.006778 5465 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:03.005462 5392 raft_consensus.cc:348] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.037257 5392 raft_consensus.cc:374] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.037560 5392 raft_consensus.cc:729] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.038206 5392 consensus_queue.cc:260] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.000609 5445 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:03.033260 5239 leader_election.cc:341] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.019271 5531 consensus_queue.cc:1035] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:03.039944 5391 tablet_bootstrap.cc:654] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.039333 5099 catalog_manager.cc:5495] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:03.041728 5198 raft_consensus.cc:2379] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 1: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:03.045673 5445 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:03.046331 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:11:03.047608 5164 leader_election.cc:341] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.049108 5393 tablet_bootstrap.cc:492] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:03.054909 5392 ts_tablet_manager.cc:1428] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.084s user 0.008s sys 0.000s
I20260929 14:11:03.055738 5392 tablet_bootstrap.cc:492] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:03.064340 5450 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:03.064851 5450 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.067365 5450 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:03.067705 5392 tablet_bootstrap.cc:654] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.068903 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:03.069983 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:03.070262 5316 leader_election.cc:341] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
W20260929 14:11:03.072081 5314 leader_election.cc:341] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.072664 5314 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:03.074136 5450 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:03.076092 5393 tablet_bootstrap.cc:654] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.085981 5242 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:03.086786 5465 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 14:11:03.088007 5391 tablet_bootstrap.cc:492] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:03.088425 5391 ts_tablet_manager.cc:1397] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.055s user 0.034s sys 0.018s
I20260929 14:11:03.091125 5391 raft_consensus.cc:348] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.091995 5391 raft_consensus.cc:374] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.092252 5391 raft_consensus.cc:729] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.093057 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:03.092971 5391 consensus_queue.cc:260] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.093796 5275 raft_consensus.cc:2379] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 1: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:03.095077 5393 tablet_bootstrap.cc:492] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:03.095469 5391 ts_tablet_manager.cc:1428] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 14:11:03.095546 5393 ts_tablet_manager.cc:1397] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.047s user 0.009s sys 0.003s
I20260929 14:11:03.096346 5391 tablet_bootstrap.cc:492] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:03.098939 5393 raft_consensus.cc:348] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.105392 5393 raft_consensus.cc:374] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.105698 5393 raft_consensus.cc:729] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.106325 5198 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.106329 5393 consensus_queue.cc:260] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.101923 5391 tablet_bootstrap.cc:654] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.102723 5392 tablet_bootstrap.cc:492] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:03.108538 5392 ts_tablet_manager.cc:1397] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.053s user 0.015s sys 0.019s
I20260929 14:11:03.110303 5445 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:03.110796 5445 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.113238 5445 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:03.114647 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:03.115935 5164 leader_election.cc:341] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.116170 5392 raft_consensus.cc:348] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.116920 5392 raft_consensus.cc:374] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.117194 5392 raft_consensus.cc:729] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.119777 5165 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:03.121521 5393 ts_tablet_manager.cc:1428] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.025s user 0.006s sys 0.001s
I20260929 14:11:03.117990 5392 consensus_queue.cc:260] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.124049 5392 ts_tablet_manager.cc:1428] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.015s user 0.001s sys 0.004s
I20260929 14:11:03.124303 5445 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 14:11:03.124939 5392 tablet_bootstrap.cc:492] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:03.126201 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:03.127321 5393 tablet_bootstrap.cc:492] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:03.128461 5419 consensus_queue.cc:1035] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:03.131057 5392 tablet_bootstrap.cc:654] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.131217 5419 consensus_queue.cc:1035] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 14:11:03.130962 5165 leader_election.cc:341] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.133709 5165 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:03.134414 5445 raft_consensus.cc:2726] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:03.154114 5392 tablet_bootstrap.cc:492] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:03.154695 5392 ts_tablet_manager.cc:1397] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.030s user 0.016s sys 0.011s
I20260929 14:11:03.157202 5392 raft_consensus.cc:348] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.157986 5392 raft_consensus.cc:374] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.159729 5393 tablet_bootstrap.cc:654] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.158238 5392 raft_consensus.cc:729] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.165845 5392 consensus_queue.cc:260] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.168649 5392 ts_tablet_manager.cc:1428] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260929 14:11:03.169764 5392 tablet_bootstrap.cc:492] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:03.180552 5445 consensus_queue.cc:1035] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:03.184114 5393 tablet_bootstrap.cc:492] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:03.184454 5393 ts_tablet_manager.cc:1397] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.000s
I20260929 14:11:03.186775 5391 tablet_bootstrap.cc:492] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:03.187192 5391 ts_tablet_manager.cc:1397] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.091s user 0.017s sys 0.020s
I20260929 14:11:03.188985 5392 tablet_bootstrap.cc:654] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.189343 5391 raft_consensus.cc:348] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.190078 5391 raft_consensus.cc:374] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.190744 5391 raft_consensus.cc:729] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.187216 5393 raft_consensus.cc:348] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.191327 5391 consensus_queue.cc:260] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.191874 5393 raft_consensus.cc:374] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.192159 5393 raft_consensus.cc:729] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.192873 5393 consensus_queue.cc:260] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.193594 5391 ts_tablet_manager.cc:1428] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 14:11:03.194389 5391 tablet_bootstrap.cc:492] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:03.195981 5393 ts_tablet_manager.cc:1428] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260929 14:11:03.199812 5391 tablet_bootstrap.cc:654] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.196805 5393 tablet_bootstrap.cc:492] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:03.223127 5393 tablet_bootstrap.cc:654] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.233045 5392 tablet_bootstrap.cc:492] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:03.233487 5392 ts_tablet_manager.cc:1397] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.064s user 0.016s sys 0.000s
I20260929 14:11:03.233778 5391 tablet_bootstrap.cc:492] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:03.234222 5391 ts_tablet_manager.cc:1397] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.040s user 0.027s sys 0.008s
I20260929 14:11:03.235387 5450 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:03.235754 5392 raft_consensus.cc:348] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.236519 5392 raft_consensus.cc:374] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.236287 5391 raft_consensus.cc:348] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.235810 5450 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.236773 5392 raft_consensus.cc:729] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.236941 5391 raft_consensus.cc:374] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.237205 5391 raft_consensus.cc:729] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.238109 5392 consensus_queue.cc:260] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.238025 5391 consensus_queue.cc:260] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.239554 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:03.240067 5392 ts_tablet_manager.cc:1428] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.006s user 0.001s sys 0.003s
I20260929 14:11:03.240249 5391 ts_tablet_manager.cc:1428] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:11:03.239996 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:03.240711 5316 leader_election.cc:341] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.240918 5392 tablet_bootstrap.cc:492] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:03.240996 5391 tablet_bootstrap.cc:492] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
W20260929 14:11:03.241472 5314 leader_election.cc:341] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.242360 5314 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:03.242499 5450 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:03.248951 5450 raft_consensus.cc:2726] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:03.249567 5450 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:03.250062 5450 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.255838 5450 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:03.255712 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:03.257499 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:03.258164 5391 tablet_bootstrap.cc:654] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.258862 5393 tablet_bootstrap.cc:492] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:03.259318 5393 ts_tablet_manager.cc:1397] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.063s user 0.012s sys 0.004s
I20260929 14:11:03.261891 5393 raft_consensus.cc:348] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.262836 5393 raft_consensus.cc:374] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.263170 5393 raft_consensus.cc:729] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.258164 5392 tablet_bootstrap.cc:654] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.265158 5445 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:03.265393 5314 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:03.263957 5393 consensus_queue.cc:260] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.266254 5568 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:03.267479 5393 ts_tablet_manager.cc:1428] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260929 14:11:03.271621 5391 tablet_bootstrap.cc:492] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:03.272085 5391 ts_tablet_manager.cc:1397] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.012s
I20260929 14:11:03.274869 5391 raft_consensus.cc:348] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.275571 5391 raft_consensus.cc:374] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.275821 5391 raft_consensus.cc:729] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.277892 5393 tablet_bootstrap.cc:492] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:03.284813 5393 tablet_bootstrap.cc:654] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.276391 5391 consensus_queue.cc:260] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.286886 5391 ts_tablet_manager.cc:1428] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20260929 14:11:03.288686 5391 tablet_bootstrap.cc:492] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:03.294250 5391 tablet_bootstrap.cc:654] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
W20260929 14:11:03.301139 5165 consensus_peers.cc:487] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd -> Peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Couldn't send request to peer c5ec116e39144881bdf1f2cd3e923cf5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:03.303085 5392 tablet_bootstrap.cc:492] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:03.303419 5392 ts_tablet_manager.cc:1397] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.063s user 0.011s sys 0.004s
I20260929 14:11:03.324841 5419 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:03.325399 5419 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.378086 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:03.379761 5418 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:03.380399 5418 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.383430 5418 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:03.385456 5242 leader_election.cc:341] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.426177 5391 tablet_bootstrap.cc:492] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:03.426661 5391 ts_tablet_manager.cc:1397] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.138s user 0.038s sys 0.061s
I20260929 14:11:03.426011 5392 raft_consensus.cc:348] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.427026 5392 raft_consensus.cc:374] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.427327 5392 raft_consensus.cc:729] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.427933 5392 consensus_queue.cc:260] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.430805 5391 raft_consensus.cc:348] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.431501 5391 raft_consensus.cc:374] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.431756 5391 raft_consensus.cc:729] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.432000 5393 tablet_bootstrap.cc:492] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:03.432438 5393 ts_tablet_manager.cc:1397] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.155s user 0.007s sys 0.005s
I20260929 14:11:03.432343 5391 consensus_queue.cc:260] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.433925 5391 ts_tablet_manager.cc:1428] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20260929 14:11:03.434741 5391 tablet_bootstrap.cc:492] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:03.434779 5393 raft_consensus.cc:348] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.435539 5393 raft_consensus.cc:374] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.435798 5393 raft_consensus.cc:729] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.436708 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:03.437208 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 51806.2us
I20260929 14:11:03.439961 5275 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.440543 5391 tablet_bootstrap.cc:654] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.449918 5447 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:03.452369 5391 tablet_bootstrap.cc:492] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:03.452787 5391 ts_tablet_manager.cc:1397] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.011s
I20260929 14:11:03.456207 5419 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:03.456108 5391 raft_consensus.cc:348] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.457027 5391 raft_consensus.cc:374] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.457406 5391 raft_consensus.cc:729] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.458279 5391 consensus_queue.cc:260] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.461026 5391 ts_tablet_manager.cc:1428] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260929 14:11:03.461035 5351 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.462595 5391 tablet_bootstrap.cc:492] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:03.464401 5474 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:03.464860 5474 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.467033 5474 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:03.468525 5391 tablet_bootstrap.cc:654] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.468849 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:03.470003 5393 consensus_queue.cc:260] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.472124 5393 ts_tablet_manager.cc:1428] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.039s user 0.007s sys 0.001s
I20260929 14:11:03.477027 5393 tablet_bootstrap.cc:492] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:03.478783 5198 raft_consensus.cc:2454] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:03.480928 5392 ts_tablet_manager.cc:1428] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.177s user 0.006s sys 0.002s
I20260929 14:11:03.485169 5242 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:03.487414 5392 tablet_bootstrap.cc:492] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:03.488844 5276 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.492935 5474 raft_consensus.cc:2781] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:03.493340 5474 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:03.493849 5474 raft_consensus.cc:3037] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.497648 5447 consensus_queue.cc:1035] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:03.498713 5445 consensus_queue.cc:1035] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:03.499290 5419 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:03.499913 5419 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.501150 5445 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:03.501598 5445 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.502789 5419 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:03.505607 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:03.533123 5568 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:03.534722 5568 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.535480 5465 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:03.535949 5465 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.536707 5568 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:03.537446 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:03.541774 5533 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:03.538805 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:03.543910 5445 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:03.544425 5314 leader_election.cc:341] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.541877 5198 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:03.544850 5533 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.545626 5316 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:03.545616 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:03.550347 5567 raft_consensus.cc:2781] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:03.550593 5533 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:03.550753 5567 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:03.551118 5567 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.542472 5465 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:03.540789 5474 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.573403 5391 tablet_bootstrap.cc:492] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:03.573920 5391 ts_tablet_manager.cc:1397] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.112s user 0.024s sys 0.014s
W20260929 14:11:03.568815 5164 leader_election.cc:341] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.575670 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:03.573032 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:03.581092 5276 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:03.581710 5393 tablet_bootstrap.cc:654] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.578083 5392 tablet_bootstrap.cc:654] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.577664 5391 raft_consensus.cc:348] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.582965 5391 raft_consensus.cc:374] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.583241 5391 raft_consensus.cc:729] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.587940 5391 consensus_queue.cc:260] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.587509 5350 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:03.612545 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:03.612747 5165 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:03.613998 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:03.614648 5201 raft_consensus.cc:2454] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:03.639914 5447 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:03.640463 5447 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:03.640877 5447 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:11:03.645596 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 45689.7us
I20260929 14:11:03.630450 5450 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:03.649667 5450 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.652737 5568 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:03.672560 5567 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.678799 5474 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:03.681782 5567 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:03.630787 5273 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:03.672323 5568 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.705549 5568 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:03.630759 5445 consensus_queue.cc:1035] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.076s
I20260929 14:11:03.612759 5164 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:03.716172 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 103549us
I20260929 14:11:03.717355 5533 raft_consensus.cc:2781] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:03.717825 5533 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:03.718218 5533 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.718312 5531 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:03.718752 5531 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.725764 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:03.726647 5350 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:03.722760 5450 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:03.732574 5391 ts_tablet_manager.cc:1428] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.158s user 0.023s sys 0.018s
I20260929 14:11:03.734475 5391 tablet_bootstrap.cc:492] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:03.730655 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:03.729485 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:03.741021 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:03.744872 5275 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:03.738382 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:03.733851 5242 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:03.743827 5531 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:03.747411 5314 leader_election.cc:341] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.756692 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:03.757326 5350 raft_consensus.cc:2454] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:03.752868 5465 raft_consensus.cc:2781] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:03.758852 5465 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:03.759267 5465 raft_consensus.cc:3037] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.759852 5391 tablet_bootstrap.cc:654] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:03.766749 5314 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:03.769621 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:03.770009 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:03.770655 5450 raft_consensus.cc:2781] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:03.771092 5450 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:03.771202 5316 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:03.771507 5450 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.772230 5568 raft_consensus.cc:2726] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:03.772162 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:03.772811 5201 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.773387 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:03.777690 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:03.778342 5350 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:03.778573 5447 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.779428 5393 tablet_bootstrap.cc:492] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:03.779847 5393 ts_tablet_manager.cc:1397] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.303s user 0.012s sys 0.043s
I20260929 14:11:03.779755 5533 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.780968 5447 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:03.782810 5164 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:03.786792 5447 raft_consensus.cc:2781] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:03.790833 5447 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:03.791214 5447 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.789395 5393 raft_consensus.cc:348] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.793300 5465 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.794024 5393 raft_consensus.cc:374] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.794384 5393 raft_consensus.cc:729] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.795238 5465 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:03.788621 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:03.794996 5393 consensus_queue.cc:260] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.798341 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:03.799753 5316 leader_election.cc:341] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.801586 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:03.802173 5199 raft_consensus.cc:3037] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.802811 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:03.803371 5275 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.807658 5393 ts_tablet_manager.cc:1428] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.028s user 0.007s sys 0.001s
I20260929 14:11:03.808084 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:03.810551 5393 tablet_bootstrap.cc:492] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:03.813879 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:03.812747 5350 raft_consensus.cc:3037] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.802194 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:03.814762 5276 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.816862 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:03.817513 5349 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.818471 5275 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:03.806583 5533 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:03.811218 5201 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
W20260929 14:11:03.830649 5164 leader_election.cc:341] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.844918 5450 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.862516 5316 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:03.863616 5450 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:03.864153 5567 raft_consensus.cc:2781] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:03.864609 5567 raft_consensus.cc:686] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:03.865589 5351 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.848536 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:03.866901 5567 consensus_queue.cc:237] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.869172 5239 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:03.870083 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:03.870692 5273 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.870618 5239 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:03.872005 5347 raft_consensus.cc:3037] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.848260 5350 raft_consensus.cc:2454] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:03.855705 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:03.872509 5419 raft_consensus.cc:2726] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:03.873713 5198 raft_consensus.cc:3037] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.874385 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:11:03.875752 5316 leader_election.cc:341] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
W20260929 14:11:03.857169 5165 leader_election.cc:341] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.857719 5533 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:03.879896 5392 tablet_bootstrap.cc:492] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:03.882004 5392 ts_tablet_manager.cc:1397] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.395s user 0.013s sys 0.043s
I20260929 14:11:03.883208 5447 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.892666 5165 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:03.895321 5397 raft_consensus.cc:2781] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:03.895993 5397 raft_consensus.cc:686] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:03.881204 5533 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.898247 5533 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:03.897136 5397 consensus_queue.cc:237] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.900936 5239 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:03.906620 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:03.911607 5239 leader_election.cc:341] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.912667 5419 raft_consensus.cc:2726] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:03.917936 5273 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:03.887063 5199 raft_consensus.cc:2454] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
W20260929 14:11:03.919721 5239 leader_election.cc:341] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.920483 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:03.880190 5349 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:03.887392 5276 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
W20260929 14:11:03.924975 5147 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260929 14:11:03.925336 5164 leader_election.cc:341] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:03.908012 5447 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:03.930023 5471 raft_consensus.cc:2741] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260929 14:11:03.942315 5239 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:03.944897 5465 raft_consensus.cc:2781] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:03.947726 5314 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:03.936916 5350 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:03.954301 5099 catalog_manager.cc:5495] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:03.954305 5391 tablet_bootstrap.cc:492] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:03.956982 5391 ts_tablet_manager.cc:1397] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.224s user 0.033s sys 0.052s
W20260929 14:11:03.951400 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 57061.2us
I20260929 14:11:03.959127 5567 raft_consensus.cc:2781] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:03.962565 5465 raft_consensus.cc:686] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:03.968941 5392 raft_consensus.cc:348] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.962945 5198 raft_consensus.cc:2454] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:03.969902 5392 raft_consensus.cc:374] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.970341 5392 raft_consensus.cc:729] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.970943 5392 consensus_queue.cc:260] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.980890 5392 ts_tablet_manager.cc:1428] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.098s user 0.003s sys 0.003s
I20260929 14:11:03.982342 5392 tablet_bootstrap.cc:492] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:03.983629 5391 raft_consensus.cc:348] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.984294 5391 raft_consensus.cc:374] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:03.984519 5391 raft_consensus.cc:729] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:03.985057 5391 consensus_queue.cc:260] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:03.967947 5465 consensus_queue.cc:237] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.009081 5391 ts_tablet_manager.cc:1428] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.052s user 0.006s sys 0.000s
I20260929 14:11:04.010447 5391 tablet_bootstrap.cc:492] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:04.013881 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:04.016651 5393 tablet_bootstrap.cc:654] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.019235 5350 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:04.020316 5164 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:04.023392 5450 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:04.025990 5567 raft_consensus.cc:686] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:04.025590 5585 raft_consensus.cc:2781] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:04.026026 5450 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.031540 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:04.035022 5392 tablet_bootstrap.cc:654] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.046962 5165 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:04.047752 5531 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:04.048326 5531 raft_consensus.cc:686] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:04.049080 5531 consensus_queue.cc:237] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.051252 5450 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:04.054747 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:04.038233 5585 raft_consensus.cc:686] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:04.029958 5567 consensus_queue.cc:237] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.077387 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:04.081602 5165 leader_election.cc:341] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:04.075274 5585 consensus_queue.cc:237] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.083307 5242 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:04.087540 5419 raft_consensus.cc:2781] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:04.089869 5419 raft_consensus.cc:686] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:04.090708 5419 consensus_queue.cc:237] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.097756 5391 tablet_bootstrap.cc:654] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.100769 5393 tablet_bootstrap.cc:492] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:04.101258 5393 ts_tablet_manager.cc:1397] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.292s user 0.016s sys 0.003s
I20260929 14:11:04.103008 5393 raft_consensus.cc:348] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.103700 5393 raft_consensus.cc:374] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.103945 5393 raft_consensus.cc:729] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.104517 5393 consensus_queue.cc:260] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.104900 5316 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:04.106120 5568 raft_consensus.cc:2726] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260929 14:11:04.109090 5165 leader_election.cc:341] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:04.109278 5393 ts_tablet_manager.cc:1428] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260929 14:11:04.110425 5393 tablet_bootstrap.cc:492] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:04.109591 5165 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:04.121001 5445 raft_consensus.cc:2726] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:04.124011 5567 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:04.123649 5098 catalog_manager.cc:5495] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:04.124465 5567 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.127933 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:04.128942 5273 raft_consensus.cc:2379] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 1: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:04.131047 5567 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:04.131258 5392 tablet_bootstrap.cc:492] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:04.131935 5392 ts_tablet_manager.cc:1397] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.150s user 0.004s sys 0.016s
I20260929 14:11:04.136827 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:04.138067 5392 raft_consensus.cc:348] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.138756 5392 raft_consensus.cc:374] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.139010 5392 raft_consensus.cc:729] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.139626 5392 consensus_queue.cc:260] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.141033 5316 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:04.141913 5567 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.149545 5392 ts_tablet_manager.cc:1428] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.017s user 0.005s sys 0.002s
I20260929 14:11:04.152261 5392 tablet_bootstrap.cc:492] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:04.162036 5567 raft_consensus.cc:2726] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:04.162361 5393 tablet_bootstrap.cc:654] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.165012 5391 tablet_bootstrap.cc:492] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:04.165184 5392 tablet_bootstrap.cc:654] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.165439 5391 ts_tablet_manager.cc:1397] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.155s user 0.018s sys 0.006s
I20260929 14:11:04.173135 5391 raft_consensus.cc:348] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.174081 5391 raft_consensus.cc:374] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.181175 5393 tablet_bootstrap.cc:492] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:04.181689 5393 ts_tablet_manager.cc:1397] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.071s user 0.009s sys 0.007s
I20260929 14:11:04.174402 5391 raft_consensus.cc:729] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.182847 5391 consensus_queue.cc:260] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.184774 5391 ts_tablet_manager.cc:1428] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20260929 14:11:04.185708 5391 tablet_bootstrap.cc:492] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:04.188908 5393 raft_consensus.cc:348] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.189821 5393 raft_consensus.cc:374] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.190179 5393 raft_consensus.cc:729] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.193182 5393 consensus_queue.cc:260] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.196658 5097 catalog_manager.cc:5495] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:04.202296 5393 ts_tablet_manager.cc:1428] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.020s user 0.007s sys 0.000s
I20260929 14:11:04.203126 5393 tablet_bootstrap.cc:492] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:04.216244 5393 tablet_bootstrap.cc:654] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.221194 5391 tablet_bootstrap.cc:654] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.226094 5392 tablet_bootstrap.cc:492] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:04.226554 5392 ts_tablet_manager.cc:1397] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.074s user 0.015s sys 0.003s
I20260929 14:11:04.229139 5392 raft_consensus.cc:348] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.230647 5392 raft_consensus.cc:374] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.231046 5392 raft_consensus.cc:729] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.233955 5392 consensus_queue.cc:260] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.236841 5392 ts_tablet_manager.cc:1428] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260929 14:11:04.237855 5392 tablet_bootstrap.cc:492] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:04.237047 5099 catalog_manager.cc:5495] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:04.239403 5099 catalog_manager.cc:5495] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:04.244244 5391 tablet_bootstrap.cc:492] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:04.244674 5391 ts_tablet_manager.cc:1397] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.059s user 0.010s sys 0.002s
I20260929 14:11:04.245687 5392 tablet_bootstrap.cc:654] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.249325 5393 tablet_bootstrap.cc:492] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:04.249801 5393 ts_tablet_manager.cc:1397] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.016s
I20260929 14:11:04.257400 5567 consensus_queue.cc:1035] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:04.252744 5393 raft_consensus.cc:348] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.259368 5393 raft_consensus.cc:374] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.259766 5393 raft_consensus.cc:729] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.259574 5098 catalog_manager.cc:5495] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:04.260452 5393 consensus_queue.cc:260] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.263443 5393 ts_tablet_manager.cc:1428] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260929 14:11:04.264500 5393 tablet_bootstrap.cc:492] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:04.269960 5391 raft_consensus.cc:348] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.270709 5391 raft_consensus.cc:374] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.270962 5391 raft_consensus.cc:729] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.271571 5391 consensus_queue.cc:260] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.272446 5585 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:04.272886 5585 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.275300 5585 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:04.275849 5393 tablet_bootstrap.cc:654] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.276072 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:04.277823 5165 leader_election.cc:341] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:04.278288 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:04.278750 5391 ts_tablet_manager.cc:1428] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.034s user 0.006s sys 0.001s
I20260929 14:11:04.278903 5351 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:04.279830 5391 tablet_bootstrap.cc:492] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:04.280370 5164 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:04.281421 5585 raft_consensus.cc:2781] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:04.281847 5585 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:04.282160 5585 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.286116 5391 tablet_bootstrap.cc:654] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.291641 5273 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.302479 5393 tablet_bootstrap.cc:492] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:04.302980 5393 ts_tablet_manager.cc:1397] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.039s user 0.010s sys 0.002s
I20260929 14:11:04.306378 5393 raft_consensus.cc:348] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.307348 5393 raft_consensus.cc:374] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.307705 5393 raft_consensus.cc:729] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.308151 5474 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:04.308626 5474 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.311309 5391 tablet_bootstrap.cc:492] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:04.310820 5097 catalog_manager.cc:5495] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:04.311851 5391 ts_tablet_manager.cc:1397] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.032s user 0.010s sys 0.000s
I20260929 14:11:04.308569 5393 consensus_queue.cc:260] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.313539 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:04.314414 5351 raft_consensus.cc:2379] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 1: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:04.315088 5391 raft_consensus.cc:348] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.315769 5391 raft_consensus.cc:374] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.316035 5391 raft_consensus.cc:729] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.316496 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:04.318419 5242 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:04.318689 5393 ts_tablet_manager.cc:1428] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.015s user 0.006s sys 0.003s
I20260929 14:11:04.319599 5393 tablet_bootstrap.cc:492] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:04.320819 5419 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.316695 5391 consensus_queue.cc:260] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.323547 5391 ts_tablet_manager.cc:1428] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260929 14:11:04.306576 5585 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.326967 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:04.327526 5351 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.333977 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:04.337466 5585 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:04.339258 5391 tablet_bootstrap.cc:492] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:04.339795 5351 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:04.340860 5164 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:04.342039 5445 raft_consensus.cc:2781] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:04.342501 5445 raft_consensus.cc:686] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:04.343261 5445 consensus_queue.cc:237] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.346002 5393 tablet_bootstrap.cc:654] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.346073 5568 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:04.346647 5568 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.348718 5568 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:04.349062 5391 tablet_bootstrap.cc:654] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.354808 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:04.355626 5200 raft_consensus.cc:2379] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 1: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:04.357633 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:04.359539 5165 leader_election.cc:341] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:04.360102 5392 tablet_bootstrap.cc:492] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:04.360715 5392 ts_tablet_manager.cc:1397] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.123s user 0.022s sys 0.028s
I20260929 14:11:04.360190 5391 tablet_bootstrap.cc:492] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:04.361045 5314 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:04.361505 5391 ts_tablet_manager.cc:1397] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.000s
I20260929 14:11:04.362097 5450 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:04.362499 5450 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.364516 5450 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:04.363343 5392 raft_consensus.cc:348] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.365392 5392 raft_consensus.cc:374] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.365636 5392 raft_consensus.cc:729] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.366087 5419 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:04.366219 5393 tablet_bootstrap.cc:492] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:04.317054 5474 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:04.366707 5393 ts_tablet_manager.cc:1397] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.047s user 0.012s sys 0.004s
I20260929 14:11:04.367223 5474 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:04.367646 5474 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.369184 5393 raft_consensus.cc:348] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.369930 5393 raft_consensus.cc:374] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.370213 5393 raft_consensus.cc:729] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.370837 5393 consensus_queue.cc:260] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.371877 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:04.372525 5200 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:04.373008 5393 ts_tablet_manager.cc:1428] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:11:04.366217 5392 consensus_queue.cc:260] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.373869 5393 tablet_bootstrap.cc:492] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:04.374337 5316 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:04.375211 5392 ts_tablet_manager.cc:1428] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260929 14:11:04.362489 5568 raft_consensus.cc:3037] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.375444 5567 raft_consensus.cc:2781] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:04.375794 5567 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:04.376068 5392 tablet_bootstrap.cc:492] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:04.376147 5567 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.377821 5391 raft_consensus.cc:348] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.378454 5391 raft_consensus.cc:374] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.378707 5391 raft_consensus.cc:729] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.379442 5391 consensus_queue.cc:260] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.381305 5474 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:04.381855 5391 ts_tablet_manager.cc:1428] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.020s user 0.005s sys 0.001s
I20260929 14:11:04.382736 5391 tablet_bootstrap.cc:492] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:04.385644 5465 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:04.386111 5465 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.389338 5568 raft_consensus.cc:2726] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:04.389751 5391 tablet_bootstrap.cc:654] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.390381 5392 tablet_bootstrap.cc:654] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.390276 5591 consensus_queue.cc:1035] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:04.391423 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:04.395026 5393 tablet_bootstrap.cc:654] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.395939 5465 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:04.371224 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:04.397801 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:04.399596 5314 leader_election.cc:341] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:04.404431 5392 tablet_bootstrap.cc:492] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:04.405015 5392 ts_tablet_manager.cc:1397] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.003s
I20260929 14:11:04.405720 5471 consensus_queue.cc:1035] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:04.407949 5393 tablet_bootstrap.cc:492] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:04.408383 5393 ts_tablet_manager.cc:1397] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.007s
I20260929 14:11:04.408440 5392 raft_consensus.cc:348] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.409168 5392 raft_consensus.cc:374] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.409440 5392 raft_consensus.cc:729] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.410035 5392 consensus_queue.cc:260] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.410704 5393 raft_consensus.cc:348] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.411338 5393 raft_consensus.cc:374] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.411590 5393 raft_consensus.cc:729] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.412017 5392 ts_tablet_manager.cc:1428] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260929 14:11:04.412159 5393 consensus_queue.cc:260] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.412832 5392 tablet_bootstrap.cc:492] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:04.413111 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:04.414647 5391 tablet_bootstrap.cc:492] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:04.415041 5391 ts_tablet_manager.cc:1397] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.007s
I20260929 14:11:04.417768 5391 raft_consensus.cc:348] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.418406 5391 raft_consensus.cc:374] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.418637 5392 tablet_bootstrap.cc:654] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.418692 5391 raft_consensus.cc:729] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.419389 5391 consensus_queue.cc:260] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.421329 5393 ts_tablet_manager.cc:1428] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260929 14:11:04.422492 5393 tablet_bootstrap.cc:492] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:04.431284 5391 ts_tablet_manager.cc:1428] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20260929 14:11:04.432489 5391 tablet_bootstrap.cc:492] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:04.445465 5585 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:04.445935 5393 tablet_bootstrap.cc:654] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.446003 5585 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.450706 5585 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:04.452266 5391 tablet_bootstrap.cc:654] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.490906 5585 consensus_queue.cc:1035] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:04.459512 5531 consensus_queue.cc:1035] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:04.465389 5393 tablet_bootstrap.cc:492] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:04.475607 5275 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.492904 5393 ts_tablet_manager.cc:1397] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.071s user 0.013s sys 0.001s
I20260929 14:11:04.495286 5393 raft_consensus.cc:348] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.495967 5393 raft_consensus.cc:374] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.496263 5393 raft_consensus.cc:729] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.496838 5393 consensus_queue.cc:260] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.483785 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:04.498181 5350 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:04.498782 5393 ts_tablet_manager.cc:1428] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260929 14:11:04.476744 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:04.499796 5393 tablet_bootstrap.cc:492] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:04.486428 5445 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:04.501012 5445 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.504242 5445 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:04.424844 5242 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:04.507287 5393 tablet_bootstrap.cc:654] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
W20260929 14:11:04.510624 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 88800us
I20260929 14:11:04.516209 5567 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.520735 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:04.521704 5200 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.521854 5391 tablet_bootstrap.cc:492] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:04.525931 5391 ts_tablet_manager.cc:1397] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.094s user 0.015s sys 0.005s
I20260929 14:11:04.533033 5164 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:04.534106 5445 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:04.534451 5445 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:04.534785 5445 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.536813 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:04.541803 5445 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.542732 5419 consensus_queue.cc:1035] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:04.543159 5242 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:04.546060 5445 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:04.548147 5164 leader_election.cc:341] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:04.550119 5391 raft_consensus.cc:348] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.551415 5391 raft_consensus.cc:374] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.552580 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:04.569880 5349 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.561127 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:04.573905 5276 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:04.576856 5391 raft_consensus.cc:729] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.578369 5391 consensus_queue.cc:260] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.579464 5591 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:04.540975 5099 catalog_manager.cc:5495] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:04.580735 5591 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.586117 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:04.586962 5276 raft_consensus.cc:2379] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 1: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:04.588482 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:04.589375 5199 raft_consensus.cc:2379] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 1: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:04.590845 5591 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:04.592669 5447 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:04.593294 5447 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.594278 5314 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:04.595639 5447 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:04.598784 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:04.599146 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:04.600956 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:04.606089 5270 raft_consensus.cc:2379] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 1: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:04.606815 5391 ts_tablet_manager.cc:1428] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.080s user 0.008s sys 0.000s
I20260929 14:11:04.608067 5391 tablet_bootstrap.cc:492] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:04.608952 5274 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.617208 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:04.603288 5591 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.635404 5465 raft_consensus.cc:2726] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:04.622278 5445 consensus_queue.cc:1035] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260929 14:11:04.627027 5567 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:04.659085 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:04.637486 5349 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:04.654000 5531 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:04.671978 5393 tablet_bootstrap.cc:492] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:04.672497 5393 ts_tablet_manager.cc:1397] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.173s user 0.041s sys 0.013s
I20260929 14:11:04.676967 5531 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.677202 5164 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:04.679322 5585 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:04.680245 5585 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:04.681330 5314 leader_election.cc:341] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:04.681584 5393 raft_consensus.cc:348] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.682271 5393 raft_consensus.cc:374] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.682524 5393 raft_consensus.cc:729] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.682924 5447 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:04.683156 5393 consensus_queue.cc:260] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.679678 5531 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:04.692658 5393 ts_tablet_manager.cc:1428] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.020s user 0.009s sys 0.001s
I20260929 14:11:04.693974 5587 consensus_queue.cc:1035] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260929 14:11:04.696241 5450 consensus_queue.cc:1035] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:04.696750 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:04.682485 5585 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:04.702242 5393 tablet_bootstrap.cc:492] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:04.681313 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:04.704845 5274 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:04.706305 5349 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
W20260929 14:11:04.707793 5164 leader_election.cc:341] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:04.708343 5392 tablet_bootstrap.cc:492] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:04.708527 5164 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:04.708761 5392 ts_tablet_manager.cc:1397] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.296s user 0.040s sys 0.080s
I20260929 14:11:04.711163 5630 consensus_queue.cc:1035] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260929 14:11:04.714846 5165 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:04.717175 5165 leader_election.cc:341] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:04.717628 5165 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:04.718788 5445 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.720346 5392 raft_consensus.cc:348] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.721547 5392 raft_consensus.cc:374] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.712061 5585 raft_consensus.cc:2781] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:04.721410 5531 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:04.723145 5585 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:04.723555 5585 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.721838 5392 raft_consensus.cc:729] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.724581 5392 consensus_queue.cc:260] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.716259 5447 raft_consensus.cc:686] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:04.731323 5391 tablet_bootstrap.cc:654] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.723883 5591 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:04.738168 5419 raft_consensus.cc:2726] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:04.737533 5630 consensus_queue.cc:1035] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:04.738658 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:04.742030 5471 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:04.742497 5471 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.735877 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:04.746729 5392 ts_tablet_manager.cc:1428] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.038s user 0.008s sys 0.000s
I20260929 14:11:04.747423 5419 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:04.747649 5392 tablet_bootstrap.cc:492] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:04.747911 5419 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.770707 5419 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:04.775645 5393 tablet_bootstrap.cc:654] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:04.779567 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:04.780113 5445 raft_consensus.cc:2726] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:04.780300 5348 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:04.784271 5471 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:04.787654 5391 tablet_bootstrap.cc:492] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:04.788030 5391 ts_tablet_manager.cc:1397] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.180s user 0.011s sys 0.002s
I20260929 14:11:04.789180 5585 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.789369 5471 consensus_queue.cc:1035] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:04.798087 5567 consensus_queue.cc:1035] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:04.762697 5447 consensus_queue.cc:237] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:04.854408 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 52217.1us
W20260929 14:11:04.863781 5165 leader_election.cc:341] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:04.867612 5165 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:04.871882 5567 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:04.872454 5567 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.854585 5585 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:04.874713 5630 raft_consensus.cc:2726] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:04.874899 5567 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:04.876674 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:04.877480 5347 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.877283 5391 raft_consensus.cc:348] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:04.877986 5391 raft_consensus.cc:374] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:04.878268 5391 raft_consensus.cc:729] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:04.882463 5391 consensus_queue.cc:260] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:04.886386 5165 leader_election.cc:341] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
W20260929 14:11:04.854725 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 62858.8us
I20260929 14:11:04.863306 5445 consensus_queue.cc:1035] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260929 14:11:04.895793 5391 ts_tablet_manager.cc:1428] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.107s user 0.013s sys 0.009s
I20260929 14:11:04.903388 5391 tablet_bootstrap.cc:492] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:04.892333 5200 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:04.894886 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:04.912134 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:04.912964 5196 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:04.922736 5393 tablet_bootstrap.cc:492] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:04.923189 5393 ts_tablet_manager.cc:1397] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.230s user 0.003s sys 0.028s
I20260929 14:11:04.920159 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:04.925607 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:04.934255 5316 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:04.935451 5591 raft_consensus.cc:2781] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:04.935930 5591 raft_consensus.cc:686] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:04.936739 5270 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.937067 5591 consensus_queue.cc:237] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:04.945626 5314 leader_election.cc:341] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:04.953330 5197 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
W20260929 14:11:04.956928 5634 log.cc:924] Time spent T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260929 14:11:04.958709 5316 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:04.959630 5568 raft_consensus.cc:2781] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:04.959982 5568 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:04.960287 5568 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:04.961500 5239 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:04.962357 5471 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:04.962754 5471 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:04.963146 5471 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:04.992856 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:04.993798 5348 raft_consensus.cc:2379] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 1: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:04.999815 5393 raft_consensus.cc:348] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.000494 5393 raft_consensus.cc:374] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:05.000782 5393 raft_consensus.cc:729] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:05.001559 5393 consensus_queue.cc:260] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:05.014592 5639 log.cc:924] Time spent T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.092s user 0.001s sys 0.000s
I20260929 14:11:05.015061 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:05.014868 5568 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.017504 5568 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:05.018043 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:05.018649 5198 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:11:05.024742 5165 leader_election.cc:341] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:05.026443 5242 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:05.026772 5393 ts_tablet_manager.cc:1428] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.103s user 0.007s sys 0.000s
I20260929 14:11:05.027565 5393 tablet_bootstrap.cc:492] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:05.032042 5392 tablet_bootstrap.cc:654] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:05.098608 5391 tablet_bootstrap.cc:654] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:05.112757 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:05.129912 5393 tablet_bootstrap.cc:654] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:05.125236 5465 raft_consensus.cc:2726] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260929 14:11:05.153467 5314 leader_election.cc:341] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:05.157049 5347 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:05.183717 5392 tablet_bootstrap.cc:492] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:05.184226 5392 ts_tablet_manager.cc:1397] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.437s user 0.016s sys 0.025s
I20260929 14:11:05.171643 5198 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:05.191478 5471 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.192461 5392 raft_consensus.cc:348] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.193249 5392 raft_consensus.cc:374] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:05.193516 5392 raft_consensus.cc:729] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:05.194168 5392 consensus_queue.cc:260] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.200977 5098 catalog_manager.cc:5495] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:05.190333 5465 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:05.208045 5316 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:05.218190 5164 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:05.215437 5471 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:05.204298 5465 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.221906 5392 ts_tablet_manager.cc:1428] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.037s user 0.008s sys 0.000s
W20260929 14:11:05.222513 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 82817us
I20260929 14:11:05.225880 5393 tablet_bootstrap.cc:492] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:05.226567 5393 ts_tablet_manager.cc:1397] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.199s user 0.011s sys 0.006s
I20260929 14:11:05.222935 5633 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:05.222890 5445 consensus_queue.cc:1035] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:05.228657 5633 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.230769 5567 raft_consensus.cc:2781] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:05.231264 5567 raft_consensus.cc:686] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:05.232033 5567 consensus_queue.cc:237] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.234079 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:05.234675 5199 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:05.241415 5393 raft_consensus.cc:348] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.242313 5393 raft_consensus.cc:374] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:05.242568 5393 raft_consensus.cc:729] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:05.243181 5393 consensus_queue.cc:260] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.246754 5392 tablet_bootstrap.cc:492] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:05.253545 5465 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:05.258634 5393 ts_tablet_manager.cc:1428] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20260929 14:11:05.259303 5641 raft_consensus.cc:2781] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:05.254082 5633 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:05.260839 5096 catalog_manager.cc:5495] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:05.263799 5393 tablet_bootstrap.cc:492] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:05.270053 5391 tablet_bootstrap.cc:492] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:05.270546 5391 ts_tablet_manager.cc:1397] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.367s user 0.028s sys 0.024s
I20260929 14:11:05.273247 5585 consensus_queue.cc:1035] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260929 14:11:05.266734 5641 raft_consensus.cc:686] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:05.262770 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:05.294921 5641 consensus_queue.cc:237] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.333694 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:05.333992 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:05.334467 5346 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:05.316821 5350 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:05.319895 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:05.332818 5568 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:05.382447 5568 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.384637 5199 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:05.401014 5568 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:05.402697 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:05.403257 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:05.404220 5199 raft_consensus.cc:2379] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 1: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:05.404431 5392 tablet_bootstrap.cc:654] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
W20260929 14:11:05.404177 5314 leader_election.cc:341] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
W20260929 14:11:05.409288 5165 leader_election.cc:341] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:05.403290 5391 raft_consensus.cc:348] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.409986 5391 raft_consensus.cc:374] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:05.410283 5391 raft_consensus.cc:729] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:05.414194 5391 consensus_queue.cc:260] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.416038 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:05.422087 5242 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:05.422776 5419 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:05.423194 5419 raft_consensus.cc:686] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:05.420620 5196 raft_consensus.cc:2454] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:05.423911 5419 consensus_queue.cc:237] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:05.432073 5147 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260929 14:11:05.432253 5471 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:05.432664 5239 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:05.432801 5471 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.434541 5316 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:05.437300 5646 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:05.437012 5567 raft_consensus.cc:2781] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:05.437717 5567 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:05.439087 5567 raft_consensus.cc:3037] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:05.439582 5350 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:05.443807 5391 ts_tablet_manager.cc:1428] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.173s user 0.013s sys 0.025s
I20260929 14:11:05.445333 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:05.446003 5197 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
I20260929 14:11:05.451000 5242 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:05.455224 5391 tablet_bootstrap.cc:492] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:05.456248 5567 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.435377 5471 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:05.459254 5465 raft_consensus.cc:2781] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:05.509089 5465 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:05.513756 5465 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:05.519204 5393 tablet_bootstrap.cc:654] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:05.507447 5567 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:05.531702 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:05.535092 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 242101us
I20260929 14:11:05.535106 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:05.537143 5198 raft_consensus.cc:3037] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:05.535915 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:05.536047 5164 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:05.545007 5656 raft_consensus.cc:2781] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:05.545408 5656 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:05.545722 5656 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:05.552693 5391 tablet_bootstrap.cc:654] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:05.577872 5392 tablet_bootstrap.cc:492] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
W20260929 14:11:05.557024 5239 leader_election.cc:341] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:05.578470 5392 ts_tablet_manager.cc:1397] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.332s user 0.046s sys 0.045s
I20260929 14:11:05.547494 5465 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.569049 5567 consensus_queue.cc:1035] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20260929 14:11:05.596283 5465 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:05.607990 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:05.608675 5199 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:11:05.609907 5314 leader_election.cc:341] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:05.624752 5445 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260929 14:11:05.656207 5646 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260929 14:11:05.656497 5314 consensus_peers.cc:487] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 -> Peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Couldn't send request to peer c5ec116e39144881bdf1f2cd3e923cf5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:05.633520 5656 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.661612 5568 consensus_queue.cc:1035] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:05.664039 5656 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:05.621470 5199 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:05.669301 5392 raft_consensus.cc:348] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.670413 5242 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:05.673650 5198 raft_consensus.cc:2454] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:05.675060 5392 raft_consensus.cc:374] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:05.675489 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:11:05.677947 5239 leader_election.cc:341] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:05.681474 5471 raft_consensus.cc:2781] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:05.681941 5471 raft_consensus.cc:686] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:05.683459 5471 consensus_queue.cc:237] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.675506 5392 raft_consensus.cc:729] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:05.694675 5392 consensus_queue.cc:260] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.724583 5531 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:05.725829 5393 tablet_bootstrap.cc:492] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:05.726321 5393 ts_tablet_manager.cc:1397] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.463s user 0.015s sys 0.010s
I20260929 14:11:05.726078 5316 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:05.729414 5392 ts_tablet_manager.cc:1428] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.149s user 0.005s sys 0.003s
I20260929 14:11:05.604998 5096 catalog_manager.cc:5495] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:05.740533 5567 raft_consensus.cc:2781] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:05.756958 5391 tablet_bootstrap.cc:492] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:05.759050 5391 ts_tablet_manager.cc:1397] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.304s user 0.042s sys 0.091s
I20260929 14:11:05.759640 5393 raft_consensus.cc:348] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.760221 5393 raft_consensus.cc:374] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:05.760483 5393 raft_consensus.cc:729] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:05.761322 5393 consensus_queue.cc:260] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.746974 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:05.764904 5567 raft_consensus.cc:686] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:05.788911 5393 ts_tablet_manager.cc:1428] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.062s user 0.007s sys 0.000s
I20260929 14:11:05.789817 5393 tablet_bootstrap.cc:492] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:05.791055 5391 raft_consensus.cc:348] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.791924 5391 raft_consensus.cc:374] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:05.792254 5391 raft_consensus.cc:729] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:05.774712 5346 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:05.793002 5391 consensus_queue.cc:260] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.797911 5391 ts_tablet_manager.cc:1428] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.038s user 0.006s sys 0.000s
I20260929 14:11:05.797995 5567 consensus_queue.cc:237] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.798741 5391 tablet_bootstrap.cc:492] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:05.799015 5392 tablet_bootstrap.cc:492] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
W20260929 14:11:05.800609 5660 log.cc:924] Time spent T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.205s user 0.004s sys 0.000s
I20260929 14:11:05.808928 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:11:05.814147 5165 leader_election.cc:341] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:05.852792 5531 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:05.858963 5531 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.869169 5393 tablet_bootstrap.cc:654] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:05.861045 5531 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:05.872725 5391 tablet_bootstrap.cc:654] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:05.877252 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:05.885353 5392 tablet_bootstrap.cc:654] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:05.881429 5098 catalog_manager.cc:5495] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
W20260929 14:11:05.880582 5165 leader_election.cc:341] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:05.887212 5568 consensus_queue.cc:1035] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:05.897989 5346 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:05.902624 5164 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:05.906242 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:05.908107 5585 raft_consensus.cc:2781] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:05.908644 5585 raft_consensus.cc:686] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:05.910229 5585 consensus_queue.cc:237] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.914618 5099 catalog_manager.cc:5495] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 2, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 2 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:05.922495 5346 raft_consensus.cc:1261] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260929 14:11:05.930588 5199 raft_consensus.cc:1261] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260929 14:11:05.932467 5419 consensus_queue.cc:1035] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260929 14:11:05.958895 5164 leader_election.cc:341] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:05.959671 5164 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:05.962230 5656 raft_consensus.cc:2726] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:05.970928 5646 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:05.971232 5465 consensus_queue.cc:1035] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:05.971424 5646 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.973321 5646 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:05.979658 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:05.981278 5392 tablet_bootstrap.cc:492] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:05.981726 5392 ts_tablet_manager.cc:1397] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.183s user 0.033s sys 0.032s
W20260929 14:11:05.984046 5242 leader_election.cc:341] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:05.983992 5392 raft_consensus.cc:348] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.984694 5392 raft_consensus.cc:374] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:05.984953 5392 raft_consensus.cc:729] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:05.985553 5392 consensus_queue.cc:260] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:05.987787 5392 ts_tablet_manager.cc:1428] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:11:05.988587 5392 tablet_bootstrap.cc:492] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:05.988520 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:05.984845 5165 consensus_peers.cc:487] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd -> Peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Couldn't send request to peer c5ec116e39144881bdf1f2cd3e923cf5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:05.992643 5391 tablet_bootstrap.cc:492] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:05.993103 5391 ts_tablet_manager.cc:1397] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.195s user 0.043s sys 0.021s
I20260929 14:11:05.993431 5531 consensus_queue.cc:1035] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:06.014501 5391 raft_consensus.cc:348] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.015205 5391 raft_consensus.cc:374] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.015453 5391 raft_consensus.cc:729] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.016047 5391 consensus_queue.cc:260] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.015861 5096 catalog_manager.cc:5495] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
W20260929 14:11:06.021514 5239 leader_election.cc:341] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.021977 5239 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:06.023253 5646 raft_consensus.cc:2726] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:06.025606 5391 ts_tablet_manager.cc:1428] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20260929 14:11:06.026424 5391 tablet_bootstrap.cc:492] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:06.038290 5393 tablet_bootstrap.cc:492] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:06.038761 5393 ts_tablet_manager.cc:1397] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.249s user 0.019s sys 0.022s
I20260929 14:11:06.043895 5391 tablet_bootstrap.cc:654] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.049834 5392 tablet_bootstrap.cc:654] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.049809 5393 raft_consensus.cc:348] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.051808 5393 raft_consensus.cc:374] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.052076 5393 raft_consensus.cc:729] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.052670 5393 consensus_queue.cc:260] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.055267 5393 ts_tablet_manager.cc:1428] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260929 14:11:06.056051 5393 tablet_bootstrap.cc:492] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:06.060321 5099 catalog_manager.cc:5495] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:06.072039 5393 tablet_bootstrap.cc:654] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.084545 5098 catalog_manager.cc:5495] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:06.089494 5391 tablet_bootstrap.cc:492] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:06.090045 5391 ts_tablet_manager.cc:1397] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.064s user 0.018s sys 0.001s
I20260929 14:11:06.090402 5419 consensus_queue.cc:1035] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:06.092613 5391 raft_consensus.cc:348] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.093643 5391 raft_consensus.cc:374] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.094038 5391 raft_consensus.cc:729] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.094707 5391 consensus_queue.cc:260] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.106853 5392 tablet_bootstrap.cc:492] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:06.107460 5392 ts_tablet_manager.cc:1397] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.119s user 0.006s sys 0.011s
I20260929 14:11:06.111804 5392 raft_consensus.cc:348] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.112610 5392 raft_consensus.cc:374] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.112912 5392 raft_consensus.cc:729] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.114248 5391 ts_tablet_manager.cc:1428] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.024s user 0.004s sys 0.001s
I20260929 14:11:06.115983 5391 tablet_bootstrap.cc:492] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:06.116523 5392 consensus_queue.cc:260] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.114734 5393 tablet_bootstrap.cc:492] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:06.117642 5393 ts_tablet_manager.cc:1397] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.016s
I20260929 14:11:06.122785 5393 raft_consensus.cc:348] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.123468 5393 raft_consensus.cc:374] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.123728 5393 raft_consensus.cc:729] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.124329 5393 consensus_queue.cc:260] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.128990 5392 ts_tablet_manager.cc:1428] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20260929 14:11:06.129895 5392 tablet_bootstrap.cc:492] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:06.136703 5393 ts_tablet_manager.cc:1428] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260929 14:11:06.137616 5393 tablet_bootstrap.cc:492] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:06.150520 5391 tablet_bootstrap.cc:654] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.155800 5392 tablet_bootstrap.cc:654] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.166819 5672 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.167461 5672 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.176234 5393 tablet_bootstrap.cc:654] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
W20260929 14:11:06.178699 5239 consensus_peers.cc:487] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 -> Peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Couldn't send request to peer d886fb1eb0474d2d9f7410c4013c45e4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:06.184710 5391 tablet_bootstrap.cc:492] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:06.184648 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:06.185209 5391 ts_tablet_manager.cc:1397] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.069s user 0.016s sys 0.008s
I20260929 14:11:06.184969 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:06.185468 5199 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:06.187613 5316 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:06.192407 5314 leader_election.cc:341] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.191777 5567 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:06.185257 5672 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:06.193440 5567 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:06.193789 5567 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:06.195763 5391 raft_consensus.cc:348] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.196480 5391 raft_consensus.cc:374] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.196723 5391 raft_consensus.cc:729] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.197309 5391 consensus_queue.cc:260] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.200345 5391 ts_tablet_manager.cc:1428] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.015s user 0.004s sys 0.002s
I20260929 14:11:06.201169 5391 tablet_bootstrap.cc:492] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:06.206804 5393 tablet_bootstrap.cc:492] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:06.207219 5391 tablet_bootstrap.cc:654] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.207957 5393 ts_tablet_manager.cc:1397] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.071s user 0.016s sys 0.001s
I20260929 14:11:06.210364 5393 raft_consensus.cc:348] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.210879 5672 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.211068 5393 raft_consensus.cc:374] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.211333 5672 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.206220 5392 tablet_bootstrap.cc:492] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:06.214823 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:06.216548 5316 leader_election.cc:341] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.213711 5392 ts_tablet_manager.cc:1397] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.084s user 0.014s sys 0.014s
I20260929 14:11:06.220228 5672 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:06.220185 5392 raft_consensus.cc:348] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.211334 5393 raft_consensus.cc:729] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.221975 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:06.225643 5392 raft_consensus.cc:374] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.226034 5392 raft_consensus.cc:729] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.226030 5393 consensus_queue.cc:260] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:06.226922 5314 leader_election.cc:341] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.227337 5314 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:06.228072 5568 raft_consensus.cc:2726] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:06.228302 5393 ts_tablet_manager.cc:1428] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.020s user 0.004s sys 0.000s
I20260929 14:11:06.229413 5393 tablet_bootstrap.cc:492] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:06.226759 5392 consensus_queue.cc:260] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.235080 5393 tablet_bootstrap.cc:654] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.240825 5392 ts_tablet_manager.cc:1428] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.023s user 0.006s sys 0.002s
I20260929 14:11:06.242082 5392 tablet_bootstrap.cc:492] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:06.242934 5567 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.245208 5567 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:06.247001 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:06.247991 5269 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:06.247067 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:06.251646 5197 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:06.263564 5391 tablet_bootstrap.cc:492] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:06.264151 5391 ts_tablet_manager.cc:1397] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.022s
I20260929 14:11:06.268153 5392 tablet_bootstrap.cc:654] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.276467 5567 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.277004 5567 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.278954 5567 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:06.279657 5393 tablet_bootstrap.cc:492] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:06.280191 5393 ts_tablet_manager.cc:1397] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.051s user 0.009s sys 0.006s
I20260929 14:11:06.279698 5391 raft_consensus.cc:348] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.280659 5391 raft_consensus.cc:374] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.280920 5391 raft_consensus.cc:729] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.281476 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:06.281611 5391 consensus_queue.cc:260] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.285691 5391 ts_tablet_manager.cc:1428] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.021s user 0.006s sys 0.001s
I20260929 14:11:06.287899 5391 tablet_bootstrap.cc:492] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
W20260929 14:11:06.282932 5314 leader_election.cc:341] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.289485 5197 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:06.291621 5316 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:06.282438 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:06.292829 5567 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:06.301820 5269 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
W20260929 14:11:06.322148 5316 leader_election.cc:341] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.321995 5393 raft_consensus.cc:348] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.323238 5393 raft_consensus.cc:374] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.323194 5316 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:06.326239 5567 raft_consensus.cc:686] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:06.332257 5567 consensus_queue.cc:237] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.335397 5672 raft_consensus.cc:2726] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:06.323693 5393 raft_consensus.cc:729] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.338783 5531 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.338207 5393 consensus_queue.cc:260] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.339234 5531 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:06.341071 5314 consensus_peers.cc:487] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 -> Peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Couldn't send request to peer c5ec116e39144881bdf1f2cd3e923cf5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:06.342675 5391 tablet_bootstrap.cc:654] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.345165 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:06.345727 5392 tablet_bootstrap.cc:492] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:06.346203 5392 ts_tablet_manager.cc:1397] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.104s user 0.012s sys 0.003s
W20260929 14:11:06.346489 5165 leader_election.cc:341] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.341154 5531 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:06.352487 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:06.341312 5393 ts_tablet_manager.cc:1428] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.061s user 0.009s sys 0.001s
I20260929 14:11:06.355290 5393 tablet_bootstrap.cc:492] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:06.357991 5096 catalog_manager.cc:5495] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
W20260929 14:11:06.366989 5164 leader_election.cc:341] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.367444 5164 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:06.368088 5531 raft_consensus.cc:2726] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:06.370113 5392 raft_consensus.cc:348] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.371222 5392 raft_consensus.cc:374] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.371523 5392 raft_consensus.cc:729] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.372382 5392 consensus_queue.cc:260] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.375823 5392 ts_tablet_manager.cc:1428] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.028s user 0.005s sys 0.001s
I20260929 14:11:06.384559 5393 tablet_bootstrap.cc:654] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.393726 5391 tablet_bootstrap.cc:492] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:06.394322 5392 tablet_bootstrap.cc:492] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:06.395720 5391 ts_tablet_manager.cc:1397] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.108s user 0.019s sys 0.009s
I20260929 14:11:06.398360 5391 raft_consensus.cc:348] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.399505 5391 raft_consensus.cc:374] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.399787 5391 raft_consensus.cc:729] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.402053 5392 tablet_bootstrap.cc:654] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.401654 5391 consensus_queue.cc:260] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.406069 5665 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.406544 5665 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.407795 5391 ts_tablet_manager.cc:1428] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260929 14:11:06.408730 5665 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:06.409759 5391 tablet_bootstrap.cc:492] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:06.410887 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:06.410887 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:06.412385 5242 leader_election.cc:341] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
W20260929 14:11:06.413555 5239 leader_election.cc:341] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.413957 5239 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:06.419605 5665 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:06.420578 5567 consensus_queue.cc:1035] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:06.424546 5391 tablet_bootstrap.cc:654] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.439139 5393 tablet_bootstrap.cc:492] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:06.439688 5393 ts_tablet_manager.cc:1397] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.085s user 0.007s sys 0.005s
I20260929 14:11:06.442317 5393 raft_consensus.cc:348] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.449182 5393 raft_consensus.cc:374] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.449666 5393 raft_consensus.cc:729] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.452668 5393 consensus_queue.cc:260] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.456599 5665 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.457155 5665 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.459235 5665 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:06.459435 5393 ts_tablet_manager.cc:1428] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.019s user 0.004s sys 0.002s
I20260929 14:11:06.464265 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:06.465795 5447 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.466686 5447 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.467500 5672 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.468026 5391 tablet_bootstrap.cc:492] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:06.467959 5672 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.468475 5391 ts_tablet_manager.cc:1397] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.026s
I20260929 14:11:06.468721 5447 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:06.469964 5672 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:06.471469 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:06.473126 5314 leader_election.cc:341] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.471493 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:06.474757 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:06.474793 5199 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:06.476244 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:06.476306 5164 leader_election.cc:341] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.477183 5316 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:06.477326 5391 raft_consensus.cc:348] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.478029 5391 raft_consensus.cc:374] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.478294 5391 raft_consensus.cc:729] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.479132 5674 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:06.479488 5674 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:06.479830 5674 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:06.477200 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:06.482199 5269 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:06.481623 5391 consensus_queue.cc:260] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.483484 5165 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:06.484071 5391 ts_tablet_manager.cc:1428] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.015s user 0.002s sys 0.003s
I20260929 14:11:06.484652 5447 raft_consensus.cc:2781] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:06.484941 5391 tablet_bootstrap.cc:492] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:06.485358 5447 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:06.486070 5393 tablet_bootstrap.cc:492] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:06.489585 5447 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:06.495239 5674 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:06.497889 5165 consensus_peers.cc:487] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd -> Peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Couldn't send request to peer c5ec116e39144881bdf1f2cd3e923cf5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:06.498898 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:06.500327 5199 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:06.499234 5447 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.505163 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:06.507357 5242 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:06.507992 5270 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:06.508730 5646 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:06.508972 5393 tablet_bootstrap.cc:654] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.511196 5392 tablet_bootstrap.cc:492] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:06.511561 5392 ts_tablet_manager.cc:1397] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.117s user 0.010s sys 0.006s
I20260929 14:11:06.520202 5391 tablet_bootstrap.cc:654] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.524498 5674 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:06.524775 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:11:06.526383 5164 leader_election.cc:341] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.526423 5270 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:06.528162 5165 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:06.531471 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:06.531721 5531 raft_consensus.cc:2781] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:06.535737 5531 raft_consensus.cc:686] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:06.535630 5392 raft_consensus.cc:348] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.536345 5392 raft_consensus.cc:374] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.536614 5392 raft_consensus.cc:729] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.536882 5531 consensus_queue.cc:237] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.537163 5392 consensus_queue.cc:260] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.527741 5447 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:06.540915 5314 leader_election.cc:341] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.545804 5392 ts_tablet_manager.cc:1428] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.034s user 0.004s sys 0.003s
I20260929 14:11:06.546650 5392 tablet_bootstrap.cc:492] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:06.550350 5393 tablet_bootstrap.cc:492] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:06.551152 5393 ts_tablet_manager.cc:1397] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.065s user 0.015s sys 0.016s
I20260929 14:11:06.552222 5391 tablet_bootstrap.cc:492] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:06.552636 5391 ts_tablet_manager.cc:1397] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.013s
I20260929 14:11:06.553763 5656 consensus_queue.cc:1035] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:06.553589 5393 raft_consensus.cc:348] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.554491 5393 raft_consensus.cc:374] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.554759 5393 raft_consensus.cc:729] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.555437 5393 consensus_queue.cc:260] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.557586 5393 ts_tablet_manager.cc:1428] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260929 14:11:06.557744 5199 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:06.559790 5316 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:06.560920 5393 tablet_bootstrap.cc:492] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:06.563426 5672 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:06.563944 5672 raft_consensus.cc:686] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:06.564095 5391 raft_consensus.cc:348] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.565805 5585 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.564808 5672 consensus_queue.cc:237] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.569644 5585 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.572043 5391 raft_consensus.cc:374] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.572333 5391 raft_consensus.cc:729] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.573236 5585 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:06.572898 5391 consensus_queue.cc:260] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.573737 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:06.575127 5391 ts_tablet_manager.cc:1428] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.022s user 0.008s sys 0.000s
W20260929 14:11:06.575179 5165 leader_election.cc:341] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.576251 5391 tablet_bootstrap.cc:492] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:06.577250 5393 tablet_bootstrap.cc:654] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.580075 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:06.584575 5164 leader_election.cc:341] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.585069 5164 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:06.588865 5391 tablet_bootstrap.cc:654] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.591080 5445 raft_consensus.cc:2726] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:06.629905 5096 catalog_manager.cc:5495] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:06.641788 5392 tablet_bootstrap.cc:654] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.665531 5393 tablet_bootstrap.cc:492] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:06.666249 5393 ts_tablet_manager.cc:1397] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.105s user 0.026s sys 0.050s
I20260929 14:11:06.669924 5445 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.670433 5445 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.672405 5445 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:06.672333 5393 raft_consensus.cc:348] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.673125 5393 raft_consensus.cc:374] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.673421 5393 raft_consensus.cc:729] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.674230 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:06.674057 5393 consensus_queue.cc:260] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:06.675959 5164 leader_election.cc:341] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.680639 5099 catalog_manager.cc:5495] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:06.682420 5393 ts_tablet_manager.cc:1428] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.013s user 0.004s sys 0.002s
I20260929 14:11:06.672960 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:06.684209 5393 tablet_bootstrap.cc:492] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:06.688154 5531 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.689335 5531 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:06.690915 5165 leader_election.cc:341] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.691425 5165 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:06.694106 5393 tablet_bootstrap.cc:654] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.698439 5445 raft_consensus.cc:2726] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:06.702867 5531 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:06.704134 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:06.705551 5164 leader_election.cc:341] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.709709 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:06.711423 5165 leader_election.cc:341] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.711843 5165 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:06.713063 5531 raft_consensus.cc:2726] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:06.728765 5674 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.731644 5674 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.734742 5674 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:06.734854 5391 tablet_bootstrap.cc:492] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:06.735378 5391 ts_tablet_manager.cc:1397] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.159s user 0.011s sys 0.018s
I20260929 14:11:06.735615 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:06.736457 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:06.737075 5316 leader_election.cc:341] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.738191 5391 raft_consensus.cc:348] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:06.738538 5314 leader_election.cc:341] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.738961 5391 raft_consensus.cc:374] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.739229 5314 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:06.739262 5391 raft_consensus.cc:729] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.742587 5674 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:06.742906 5391 consensus_queue.cc:260] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.747679 5391 ts_tablet_manager.cc:1428] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.012s user 0.007s sys 0.002s
I20260929 14:11:06.748720 5391 tablet_bootstrap.cc:492] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:06.754266 5393 tablet_bootstrap.cc:492] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:06.757649 5393 ts_tablet_manager.cc:1397] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.074s user 0.010s sys 0.005s
I20260929 14:11:06.760905 5392 tablet_bootstrap.cc:492] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:06.761369 5392 ts_tablet_manager.cc:1397] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.215s user 0.014s sys 0.050s
I20260929 14:11:06.765530 5270 raft_consensus.cc:3037] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:06.766988 5392 raft_consensus.cc:348] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.767660 5392 raft_consensus.cc:374] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.767890 5392 raft_consensus.cc:729] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.768450 5392 consensus_queue.cc:260] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.772388 5391 tablet_bootstrap.cc:654] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.775136 5392 ts_tablet_manager.cc:1428] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260929 14:11:06.775966 5445 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.776826 5445 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.779644 5393 raft_consensus.cc:348] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.780298 5393 raft_consensus.cc:374] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.780512 5392 tablet_bootstrap.cc:492] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:06.781032 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:06.782318 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:06.782922 5351 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:06.784132 5164 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:06.784747 5445 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:06.784945 5531 raft_consensus.cc:2781] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:06.780539 5393 raft_consensus.cc:729] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.785311 5531 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:06.785704 5531 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:06.785874 5393 consensus_queue.cc:260] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:06.787478 5165 leader_election.cc:341] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.788720 5393 ts_tablet_manager.cc:1428] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.031s user 0.006s sys 0.000s
I20260929 14:11:06.790639 5393 tablet_bootstrap.cc:492] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:06.793174 5392 tablet_bootstrap.cc:654] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
W20260929 14:11:06.795591 5147 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260929 14:11:06.803874 5656 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.804409 5656 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.806565 5656 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:06.811161 5531 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.811669 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:06.813135 5165 leader_election.cc:341] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.814126 5531 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:06.816453 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:11:06.826114 5165 leader_election.cc:341] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.845284 5419 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.845940 5419 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.846755 5391 tablet_bootstrap.cc:492] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:06.847172 5391 ts_tablet_manager.cc:1397] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.099s user 0.012s sys 0.002s
I20260929 14:11:06.848048 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:06.848994 5419 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:06.849190 5349 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:06.853440 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:06.852703 5567 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.854300 5567 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.856366 5567 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:06.858940 5656 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.860072 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:06.860186 5656 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.861454 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:06.859581 5164 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:06.862790 5447 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:06.863297 5447 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:06.860671 5346 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:06.863708 5447 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:11:06.863405 5314 leader_election.cc:341] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.866014 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:06.866096 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:06.865384 5672 consensus_queue.cc:1035] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:06.866703 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:06.869526 5275 raft_consensus.cc:2454] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:06.869786 5567 consensus_queue.cc:1035] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:06.871109 5672 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.871294 5392 tablet_bootstrap.cc:492] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:06.871755 5392 ts_tablet_manager.cc:1397] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.096s user 0.021s sys 0.006s
I20260929 14:11:06.872493 5672 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.874594 5672 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:06.875169 5656 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:06.872223 5165 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:06.877015 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:06.877892 5239 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:06.878094 5656 raft_consensus.cc:2781] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:06.878598 5656 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:06.878916 5656 raft_consensus.cc:3037] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:06.879340 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:06.879729 5665 raft_consensus.cc:2726] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:06.880018 5269 raft_consensus.cc:2454] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:06.882220 5314 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:06.882794 5316 leader_election.cc:341] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.884681 5672 raft_consensus.cc:2781] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:06.886405 5672 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:06.888265 5672 raft_consensus.cc:3037] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:06.888645 5393 tablet_bootstrap.cc:654] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.888870 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:06.892676 5164 leader_election.cc:341] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.893478 5346 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:06.890805 5447 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:06.894831 5316 leader_election.cc:341] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.895258 5316 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:06.895613 5164 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:06.896719 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:06.897091 5445 raft_consensus.cc:2781] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:06.897553 5445 raft_consensus.cc:686] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:06.898195 5445 consensus_queue.cc:237] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.902381 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:06.896103 5567 raft_consensus.cc:2726] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:06.902945 5346 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:06.903724 5391 raft_consensus.cc:348] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.904273 5391 raft_consensus.cc:374] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.904510 5391 raft_consensus.cc:729] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
W20260929 14:11:06.904753 5165 leader_election.cc:341] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.905098 5391 consensus_queue.cc:260] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.913771 5269 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:06.915907 5392 raft_consensus.cc:348] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.917834 5392 raft_consensus.cc:374] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:06.918079 5392 raft_consensus.cc:729] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:06.929666 5391 ts_tablet_manager.cc:1428] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.082s user 0.007s sys 0.000s
I20260929 14:11:06.930809 5391 tablet_bootstrap.cc:492] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:06.895695 5447 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:06.954442 5391 tablet_bootstrap.cc:654] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:06.964921 5392 consensus_queue.cc:260] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.953387 5346 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:06.971975 5674 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.972477 5674 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.977375 5672 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.979293 5672 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:06.979457 5674 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:06.981353 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:06.981694 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:06.982275 5197 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:06.983904 5098 catalog_manager.cc:5495] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
W20260929 14:11:06.985865 5316 leader_election.cc:341] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:06.986637 5419 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:06.987396 5393 tablet_bootstrap.cc:492] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:06.987035 5419 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:06.987777 5393 ts_tablet_manager.cc:1397] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.197s user 0.013s sys 0.020s
I20260929 14:11:06.988672 5164 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:06.990185 5447 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:06.991658 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:06.992702 5447 raft_consensus.cc:686] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:06.993897 5419 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:06.994441 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:06.995046 5270 raft_consensus.cc:3037] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:06.995771 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:06.996562 5316 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:06.997773 5314 leader_election.cc:341] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:07.000269 5674 raft_consensus.cc:2781] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:07.000623 5674 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:07.000926 5674 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:07.004067 5270 raft_consensus.cc:2454] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
W20260929 14:11:07.006843 5164 consensus_peers.cc:487] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd -> Peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Couldn't send request to peer d886fb1eb0474d2d9f7410c4013c45e4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:07.009195 5568 consensus_queue.cc:1035] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:07.009593 5314 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:07.011642 5568 raft_consensus.cc:2781] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:07.012178 5568 raft_consensus.cc:686] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:06.993502 5447 consensus_queue.cc:237] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.020566 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:07.021471 5197 raft_consensus.cc:2379] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 1: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:07.023411 5242 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:07.026002 5392 ts_tablet_manager.cc:1428] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.154s user 0.008s sys 0.000s
I20260929 14:11:07.027323 5392 tablet_bootstrap.cc:492] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:07.027724 5665 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:07.028517 5568 consensus_queue.cc:237] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.036765 5393 raft_consensus.cc:348] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.037647 5393 raft_consensus.cc:374] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:07.037921 5393 raft_consensus.cc:729] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:07.038504 5393 consensus_queue.cc:260] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.014011 5656 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.059111 5393 ts_tablet_manager.cc:1428] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.071s user 0.009s sys 0.000s
I20260929 14:11:07.059883 5393 tablet_bootstrap.cc:492] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:07.069824 5656 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:07.065354 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 55868.6us
I20260929 14:11:07.071542 5665 raft_consensus.cc:2726] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:07.072587 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:07.073210 5273 raft_consensus.cc:3037] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:07.095412 5392 tablet_bootstrap.cc:654] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:07.098443 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:11:07.101920 5164 leader_election.cc:341] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:07.081745 5674 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.109673 5716 consensus_queue.cc:1035] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:07.109890 5273 raft_consensus.cc:2454] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:07.122016 5391 tablet_bootstrap.cc:492] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:07.112038 5445 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:07.111792 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:07.123310 5674 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:07.123509 5200 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:07.125835 5445 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.128018 5391 ts_tablet_manager.cc:1397] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.192s user 0.019s sys 0.013s
I20260929 14:11:07.121055 5393 tablet_bootstrap.cc:654] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:07.133169 5445 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:07.137390 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:07.139600 5165 leader_election.cc:341] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:07.142309 5165 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:07.143004 5447 raft_consensus.cc:2781] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:07.144759 5447 raft_consensus.cc:686] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:07.145056 5419 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:07.145792 5419 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.145435 5447 consensus_queue.cc:237] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.149103 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:07.151158 5419 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:07.149127 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:07.152660 5349 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:07.157122 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:07.157889 5164 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:07.159125 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:07.160179 5656 raft_consensus.cc:2781] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:07.160543 5656 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:07.160864 5656 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:11:07.163576 5314 leader_election.cc:341] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:07.164958 5391 raft_consensus.cc:348] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.165117 5242 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:07.168825 5391 raft_consensus.cc:374] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:07.169128 5391 raft_consensus.cc:729] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:07.169864 5391 consensus_queue.cc:260] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.172587 5646 raft_consensus.cc:2726] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:07.178155 5200 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:07.179333 5316 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:07.180367 5674 raft_consensus.cc:2781] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:07.180825 5674 raft_consensus.cc:686] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:07.175511 5099 catalog_manager.cc:5495] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:07.186744 5445 consensus_queue.cc:1035] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:07.189177 5391 ts_tablet_manager.cc:1428] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.061s user 0.008s sys 0.003s
I20260929 14:11:07.190616 5391 tablet_bootstrap.cc:492] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:07.182173 5674 consensus_queue.cc:237] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.185763 5096 catalog_manager.cc:5495] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:07.205161 5392 tablet_bootstrap.cc:492] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:07.357347 5392 ts_tablet_manager.cc:1397] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.330s user 0.041s sys 0.064s
I20260929 14:11:07.207386 5568 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:07.605581 5392 raft_consensus.cc:348] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.643388 5392 raft_consensus.cc:374] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:07.643782 5392 raft_consensus.cc:729] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:07.257814 5656 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.785598 5393 tablet_bootstrap.cc:492] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:07.786304 5393 ts_tablet_manager.cc:1397] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.727s user 0.028s sys 0.011s
I20260929 14:11:07.644640 5392 consensus_queue.cc:260] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.605455 5568 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.794579 5392 ts_tablet_manager.cc:1428] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.437s user 0.006s sys 0.002s
I20260929 14:11:07.795594 5392 tablet_bootstrap.cc:492] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:07.832078 5568 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:07.835917 5391 tablet_bootstrap.cc:654] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:07.841107 5392 tablet_bootstrap.cc:654] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:07.841295 5393 raft_consensus.cc:348] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.842062 5393 raft_consensus.cc:374] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:07.842331 5393 raft_consensus.cc:729] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:07.843355 5656 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:07.844736 5393 consensus_queue.cc:260] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.846746 5393 ts_tablet_manager.cc:1428] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.060s user 0.005s sys 0.000s
I20260929 14:11:07.847651 5393 tablet_bootstrap.cc:492] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:07.855762 5391 tablet_bootstrap.cc:492] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:07.856230 5391 ts_tablet_manager.cc:1397] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.666s user 0.020s sys 0.039s
I20260929 14:11:07.857013 5393 tablet_bootstrap.cc:654] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:07.859360 5391 raft_consensus.cc:348] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.860139 5391 raft_consensus.cc:374] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:07.860381 5391 raft_consensus.cc:729] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:07.860872 5391 consensus_queue.cc:260] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.862763 5391 ts_tablet_manager.cc:1428] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260929 14:11:07.863260 5393 tablet_bootstrap.cc:492] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:07.863675 5393 ts_tablet_manager.cc:1397] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20260929 14:11:07.863735 5391 tablet_bootstrap.cc:492] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:07.867079 5392 tablet_bootstrap.cc:492] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:07.867591 5392 ts_tablet_manager.cc:1397] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.072s user 0.015s sys 0.003s
I20260929 14:11:07.867053 5393 raft_consensus.cc:348] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.867892 5393 raft_consensus.cc:374] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:07.868161 5393 raft_consensus.cc:729] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:07.868880 5393 consensus_queue.cc:260] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.870312 5391 tablet_bootstrap.cc:654] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:07.870239 5392 raft_consensus.cc:348] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.871227 5392 raft_consensus.cc:374] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:07.871502 5392 raft_consensus.cc:729] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:07.872156 5392 consensus_queue.cc:260] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.874532 5392 ts_tablet_manager.cc:1428] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260929 14:11:07.875397 5392 tablet_bootstrap.cc:492] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:07.881099 5392 tablet_bootstrap.cc:654] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:07.882602 5391 tablet_bootstrap.cc:492] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:07.883055 5391 ts_tablet_manager.cc:1397] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20260929 14:11:07.885463 5391 raft_consensus.cc:348] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.886161 5391 raft_consensus.cc:374] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:07.886421 5391 raft_consensus.cc:729] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:07.886986 5391 consensus_queue.cc:260] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.889474 5391 ts_tablet_manager.cc:1428] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260929 14:11:07.890897 5391 tablet_bootstrap.cc:492] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:07.891959 5656 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:07.892724 5656 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.895433 5656 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:07.907881 5726 log.cc:924] Time spent T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.217s user 0.001s sys 0.001s
I20260929 14:11:07.908558 5391 tablet_bootstrap.cc:654] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:07.917740 5393 ts_tablet_manager.cc:1428] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.054s user 0.007s sys 0.000s
I20260929 14:11:07.919974 5393 tablet_bootstrap.cc:492] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
W20260929 14:11:07.922029 5240 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.017s sys 0.030s
W20260929 14:11:07.922325 5240 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.589s user 0.017s sys 0.030s
I20260929 14:11:07.933941 5391 tablet_bootstrap.cc:492] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:07.935071 5391 ts_tablet_manager.cc:1397] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.044s user 0.013s sys 0.005s
I20260929 14:11:07.936347 5393 tablet_bootstrap.cc:654] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:07.939213 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:07.939793 5349 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:07.946261 5349 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:07.950187 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:07.952485 5393 tablet_bootstrap.cc:492] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:07.953115 5393 ts_tablet_manager.cc:1397] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.000s
I20260929 14:11:07.955363 5164 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:07.956260 5743 raft_consensus.cc:2781] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:07.955971 5393 raft_consensus.cc:348] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.956761 5743 raft_consensus.cc:686] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:07.958000 5743 consensus_queue.cc:237] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.961781 5740 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:07.956791 5393 raft_consensus.cc:374] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:07.962607 5393 raft_consensus.cc:729] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:07.963506 5393 consensus_queue.cc:260] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:07.963912 5740 consensus_queue.cc:1035] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:07.965687 5393 ts_tablet_manager.cc:1428] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260929 14:11:07.966631 5393 tablet_bootstrap.cc:492] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:07.972047 5393 tablet_bootstrap.cc:654] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
W20260929 14:11:07.984764 5725 log.cc:924] Time spent T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.199s user 0.002s sys 0.001s
W20260929 14:11:07.985718 5147 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260929 14:11:08.009977 5392 tablet_bootstrap.cc:492] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:08.010578 5392 ts_tablet_manager.cc:1397] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.135s user 0.014s sys 0.016s
I20260929 14:11:08.012233 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:08.018267 5242 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.013s sys 0.032s
W20260929 14:11:08.018677 5242 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.685s user 0.013s sys 0.032s
I20260929 14:11:08.022786 5716 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:08.023262 5716 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:08.025981 5314 leader_election.cc:341] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:08.026723 5716 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:11:08.027252 5314 consensus_peers.cc:487] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 -> Peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Couldn't send request to peer c5ec116e39144881bdf1f2cd3e923cf5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:08.028174 5716 consensus_queue.cc:1035] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:08.031664 5392 raft_consensus.cc:348] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.061643 5275 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.295145 5392 raft_consensus.cc:374] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:08.295533 5392 raft_consensus.cc:729] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:08.075996 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:08.075941 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:08.298456 5392 consensus_queue.cc:260] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.299368 5270 raft_consensus.cc:2379] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 1: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:08.091411 5393 tablet_bootstrap.cc:492] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:08.300189 5393 ts_tablet_manager.cc:1397] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.334s user 0.020s sys 0.000s
I20260929 14:11:08.300734 5392 ts_tablet_manager.cc:1428] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.290s user 0.009s sys 0.000s
I20260929 14:11:08.301726 5392 tablet_bootstrap.cc:492] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:08.302603 5393 raft_consensus.cc:348] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.303287 5393 raft_consensus.cc:374] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:08.303512 5393 raft_consensus.cc:729] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:08.304057 5393 consensus_queue.cc:260] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.099472 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:08.114679 5737 consensus_queue.cc:1035] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:08.104774 5728 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:08.123149 5729 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:08.306715 5728 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.306787 5729 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.308804 5728 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:08.309417 5729 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:08.123574 5730 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:08.310151 5730 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.312368 5730 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:08.188941 5739 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:08.313459 5739 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.315724 5739 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:08.210884 5674 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:08.316613 5674 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.318850 5674 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:08.216187 5732 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:08.319460 5732 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.321545 5732 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:08.214841 5731 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:08.223395 5758 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:08.066118 5391 raft_consensus.cc:348] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.227365 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:08.322713 5199 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:08.322255 5731 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.229697 5196 raft_consensus.cc:3037] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.324617 5731 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:08.231081 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:08.251772 5672 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:08.325908 5672 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.327888 5672 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:08.329353 5096 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52726 (request call id 81) took 1159 ms. Trace:
I20260929 14:11:08.322283 5391 raft_consensus.cc:374] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:08.332978 5391 raft_consensus.cc:729] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:08.333585 5391 consensus_queue.cc:260] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.334468 5347 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.322304 5758 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.329722 5096 rpcz_store.cc:276] 0929 14:11:07.172287 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:07.178751 (+ 6464us) service_pool.cc:227] Handling call
0929 14:11:08.329267 (+1150516us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:11:07.974516 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:11:07.975039 (+ 523us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:11:07.975073 (+ 34us) write_op.cc:435] Acquired schema lock
0929 14:11:07.975090 (+ 17us) tablet.cc:660] Decoding operations
0929 14:11:07.975431 (+ 341us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:11:07.975477 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0929 14:11:07.975500 (+ 23us) tablet.cc:683] Acquiring locks for 1 operations
0929 14:11:07.975679 (+ 179us) tablet.cc:699] Row locks acquired
0929 14:11:07.975696 (+ 17us) write_op.cc:260] PREPARE: finished
0929 14:11:07.975813 (+ 117us) write_op.cc:270] Start()
0929 14:11:07.975903 (+ 90us) write_op.cc:276] Timestamp: P: 1790691067975779 usec, L: 0
0929 14:11:07.975920 (+ 17us) op_driver.cc:331] REPLICATION: starting
0929 14:11:07.976434 (+ 514us) log.cc:842] Serialized 537 byte log entry
0929 14:11:08.074072 (+ 97638us) op_driver.cc:444] REPLICATION: finished
0929 14:11:08.263144 (+189072us) write_op.cc:301] APPLY: starting
0929 14:11:08.263209 (+ 65us) tablet.cc:1365] starting BulkCheckPresence
0929 14:11:08.263305 (+ 96us) tablet.cc:1368] finished BulkCheckPresence
0929 14:11:08.263396 (+ 91us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:11:08.263582 (+ 186us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:11:08.263719 (+ 137us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0929 14:11:08.263762 (+ 43us) write_op.cc:312] APPLY: finished
0929 14:11:08.263937 (+ 175us) log.cc:842] Serialized 22 byte log entry
0929 14:11:08.264097 (+ 160us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:08.264178 (+ 81us) write_op.cc:454] Released schema lock
0929 14:11:08.264211 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0929 14:11:08.264442 (+ 231us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:08.264470 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":47157,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":188829,"apply.run_cpu_time_us":1523,"apply.run_wall_time_us":1523,"mutex_wait_us":772211,"num_ops":1,"prepare.queue_time_us":772679,"prepare.run_cpu_time_us":2150,"prepare.run_wall_time_us":2150,"replication_time_us":98058}]]}
I20260929 14:11:08.339793 5758 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:08.340965 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:08.341995 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:08.343425 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:08.344594 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:08.345777 5314 leader_election.cc:341] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
W20260929 14:11:08.346369 5314 leader_election.cc:341] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:08.258766 5759 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
W20260929 14:11:08.271136 5239 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.939s user 0.008s sys 0.004s
I20260929 14:11:08.273541 5665 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:11:08.356587 5239 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.085s user 0.008s sys 0.004s
W20260929 14:11:08.147564 5165 consensus_peers.cc:487] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd -> Peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Couldn't send request to peer c5ec116e39144881bdf1f2cd3e923cf5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:08.289389 5568 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
W20260929 14:11:08.359493 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 212035us
I20260929 14:11:08.359606 5568 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.361747 5568 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:08.293255 5567 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:08.365185 5759 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.365311 5665 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.367436 5665 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:08.367697 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:08.372560 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:08.373140 5759 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:08.365464 5567 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.379647 5567 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:08.380702 5391 ts_tablet_manager.cc:1428] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.445s user 0.010s sys 0.000s
I20260929 14:11:08.381682 5391 tablet_bootstrap.cc:492] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:08.381678 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:08.383785 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 93207.7us
I20260929 14:11:08.385118 5165 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:08.386198 5165 leader_election.cc:341] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:08.387744 5739 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:08.388509 5391 tablet_bootstrap.cc:654] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:08.390125 5567 consensus_queue.cc:1035] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:08.390622 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:08.390909 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 85813.3us
I20260929 14:11:08.392675 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:08.394603 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:08.394860 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:08.411080 5391 tablet_bootstrap.cc:492] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:08.411533 5391 ts_tablet_manager.cc:1397] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.003s
I20260929 14:11:08.413865 5391 raft_consensus.cc:348] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.414547 5391 raft_consensus.cc:374] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:08.414801 5391 raft_consensus.cc:729] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:08.415395 5391 consensus_queue.cc:260] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.416723 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:08.419512 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:08.420809 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:08.421495 5342 raft_consensus.cc:3037] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.356609 5392 tablet_bootstrap.cc:654] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:08.450037 5393 ts_tablet_manager.cc:1428] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.150s user 0.008s sys 0.000s
I20260929 14:11:08.450985 5393 tablet_bootstrap.cc:492] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:08.458662 5242 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:08.459412 5419 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:08.448230 5096 catalog_manager.cc:5495] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:08.464395 5271 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.466066 5391 ts_tablet_manager.cc:1428] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.054s user 0.011s sys 0.001s
I20260929 14:11:08.473582 5741 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:08.453490 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 63293.6us
I20260929 14:11:08.474143 5741 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.473605 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:08.478853 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:08.489046 5270 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.496840 5242 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:08.507298 5242 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:08.514021 5391 tablet_bootstrap.cc:492] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:08.515398 5316 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:08.516595 5393 tablet_bootstrap.cc:654] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:08.517117 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:08.519203 5165 leader_election.cc:341] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:08.509691 5465 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:08.512799 5740 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:08.523869 5740 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.529384 5740 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:08.510927 5741 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:08.531291 5730 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:08.538847 5730 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:08.539216 5730 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.533156 5316 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:08.534979 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:08.555411 5346 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:08.561656 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:08.558012 5567 raft_consensus.cc:2726] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:08.552764 5099 catalog_manager.cc:5495] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:08.601423 5099 catalog_manager.cc:5495] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
W20260929 14:11:08.601333 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 68272.4us
I20260929 14:11:08.607076 5567 consensus_queue.cc:1035] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:08.607585 5392 tablet_bootstrap.cc:492] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
W20260929 14:11:08.607359 5165 leader_election.cc:341] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:08.608683 5392 ts_tablet_manager.cc:1397] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.307s user 0.015s sys 0.025s
I20260929 14:11:08.613539 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:08.614255 5197 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:08.616755 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:08.620319 5164 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:08.621407 5164 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:08.627286 5763 consensus_queue.cc:1035] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260929 14:11:08.628573 5739 consensus_queue.cc:1035] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:08.629316 5656 consensus_queue.cc:1035] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:08.625115 5741 raft_consensus.cc:2781] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:08.623353 5740 raft_consensus.cc:2726] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:08.640230 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:08.643155 5164 leader_election.cc:341] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:08.643565 5164 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:08.642200 5741 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:08.645602 5741 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.649255 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:08.649717 5737 raft_consensus.cc:2726] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260929 14:11:08.654632 5165 consensus_peers.cc:487] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd -> Peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Couldn't send request to peer c5ec116e39144881bdf1f2cd3e923cf5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:08.662652 5391 tablet_bootstrap.cc:654] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:08.659950 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:08.668336 5165 leader_election.cc:341] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:08.651782 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:08.678146 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:08.680182 5665 raft_consensus.cc:2726] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:08.681046 5393 tablet_bootstrap.cc:492] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:08.681525 5393 ts_tablet_manager.cc:1397] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.231s user 0.011s sys 0.012s
I20260929 14:11:08.682647 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:08.683228 5199 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:08.685009 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:08.684942 5392 raft_consensus.cc:348] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.685766 5392 raft_consensus.cc:374] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:08.686050 5392 raft_consensus.cc:729] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:08.685444 5393 raft_consensus.cc:348] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.686542 5393 raft_consensus.cc:374] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:08.686863 5393 raft_consensus.cc:729] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:08.686650 5392 consensus_queue.cc:260] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.687654 5393 consensus_queue.cc:260] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.689163 5392 ts_tablet_manager.cc:1428] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.080s user 0.005s sys 0.001s
I20260929 14:11:08.679176 5350 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:08.692663 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:08.699422 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:08.698657 5316 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:08.702551 5730 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.703025 5392 tablet_bootstrap.cc:492] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:08.714807 5759 raft_consensus.cc:2781] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:08.715305 5759 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:08.715685 5759 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.721489 5393 ts_tablet_manager.cc:1428] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20260929 14:11:08.723088 5393 tablet_bootstrap.cc:492] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:08.724434 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:11:08.729809 5314 leader_election.cc:341] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:08.724526 5730 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:11:08.732074 5772 log.cc:924] Time spent T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260929 14:11:08.734678 5316 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:08.769155 5728 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:08.769868 5728 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.773407 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:08.773676 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:08.774787 5728 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:08.775246 5239 leader_election.cc:341] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:08.748785 5759 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.777181 5730 raft_consensus.cc:2726] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:08.764288 5737 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:08.781424 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 50018.1us
I20260929 14:11:08.782228 5316 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:08.782994 5730 raft_consensus.cc:2726] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:08.783383 5316 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:08.788705 5164 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:08.790758 5763 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:08.790191 5730 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:08.801062 5316 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:08.802021 5568 raft_consensus.cc:2781] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:08.802385 5393 tablet_bootstrap.cc:654] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:08.802425 5568 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:08.806170 5392 tablet_bootstrap.cc:654] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:08.802840 5568 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.799960 5763 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:08.809770 5763 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.785625 5737 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.821486 5316 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:08.824755 5762 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:08.825173 5567 raft_consensus.cc:2726] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:08.827498 5391 tablet_bootstrap.cc:492] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:08.828392 5391 ts_tablet_manager.cc:1397] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.315s user 0.014s sys 0.038s
I20260929 14:11:08.825368 5762 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.818583 5759 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:08.834990 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:08.835398 5762 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:08.827436 5196 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 0.
W20260929 14:11:08.837595 5314 leader_election.cc:341] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:08.837852 5737 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:08.839098 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:08.828303 5741 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.841302 5741 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:08.840577 5316 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:08.845945 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:08.846634 5567 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:08.847553 5165 leader_election.cc:341] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:08.847445 5567 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:08.848850 5165 leader_election.cc:341] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:08.851667 5759 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:08.843223 5568 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.854110 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:08.863066 5391 raft_consensus.cc:348] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.863870 5391 raft_consensus.cc:374] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:08.864135 5391 raft_consensus.cc:729] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:08.861133 5316 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:08.864984 5730 raft_consensus.cc:2726] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:08.864732 5391 consensus_queue.cc:260] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.869812 5568 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:08.870232 5391 ts_tablet_manager.cc:1428] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.042s user 0.004s sys 0.003s
I20260929 14:11:08.871093 5391 tablet_bootstrap.cc:492] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:08.871054 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:08.871115 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:08.871786 5271 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.872066 5567 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:11:08.873255 5314 leader_election.cc:341] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:08.873752 5242 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:08.862229 5276 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:08.892163 5665 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:08.894891 5665 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:08.895295 5665 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.896307 5271 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:08.884168 5763 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.898644 5314 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:08.899423 5730 raft_consensus.cc:2781] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:08.899880 5730 raft_consensus.cc:686] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:08.900569 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:08.900709 5730 consensus_queue.cc:237] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:08.902091 5165 leader_election.cc:341] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:08.903258 5393 tablet_bootstrap.cc:492] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:08.903702 5393 ts_tablet_manager.cc:1397] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.181s user 0.012s sys 0.008s
I20260929 14:11:08.907856 5763 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:08.910871 5165 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:08.910526 5393 raft_consensus.cc:348] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.913676 5393 raft_consensus.cc:374] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:08.913946 5393 raft_consensus.cc:729] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:08.914662 5393 consensus_queue.cc:260] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.916617 5393 ts_tablet_manager.cc:1428] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.013s user 0.007s sys 0.000s
I20260929 14:11:08.917392 5393 tablet_bootstrap.cc:492] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:08.918092 5741 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:08.918557 5741 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:08.919015 5741 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.926684 5392 tablet_bootstrap.cc:492] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:08.927206 5392 ts_tablet_manager.cc:1397] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.224s user 0.011s sys 0.001s
I20260929 14:11:08.930374 5392 raft_consensus.cc:348] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.931043 5392 raft_consensus.cc:374] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:08.931294 5392 raft_consensus.cc:729] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:08.931865 5392 consensus_queue.cc:260] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.934440 5392 ts_tablet_manager.cc:1428] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260929 14:11:08.935299 5392 tablet_bootstrap.cc:492] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:08.940687 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:08.940768 5665 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.941635 5197 raft_consensus.cc:2379] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 1: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:08.941874 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:08.946085 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:08.947049 5200 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.945204 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:08.949916 5201 raft_consensus.cc:2454] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:08.951057 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:08.951925 5347 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.949270 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:08.955849 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:08.956605 5199 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.958892 5665 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:08.976361 5568 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:08.983153 5316 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:08.983016 5568 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:08.984395 5316 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:08.981760 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:08.986627 5567 raft_consensus.cc:2726] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 14:11:08.987222 5393 tablet_bootstrap.cc:654] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
W20260929 14:11:08.988758 5164 leader_election.cc:341] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:08.991443 5164 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:08.988956 5730 raft_consensus.cc:2781] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:08.995481 5730 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:08.995857 5730 raft_consensus.cc:3037] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:08.995932 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:08.996367 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:08.989979 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:09.009572 5344 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:09.015429 5346 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:09.000793 5737 raft_consensus.cc:2726] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:08.998759 5568 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:09.047343 5345 raft_consensus.cc:2379] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 1: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:09.055764 5392 tablet_bootstrap.cc:654] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:09.065104 5096 catalog_manager.cc:5495] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
W20260929 14:11:09.075052 5297 debug-util.cc:398] Leaking SignalData structure 0x7b08001681a0 after lost signal to thread 4044
W20260929 14:11:09.076004 5297 debug-util.cc:398] Leaking SignalData structure 0x7b0800168880 after lost signal to thread 5133
I20260929 14:11:09.076381 5164 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:09.077139 5297 debug-util.cc:398] Leaking SignalData structure 0x7b08000ea000 after lost signal to thread 5225
W20260929 14:11:09.078199 5297 debug-util.cc:398] Leaking SignalData structure 0x7b080009c980 after lost signal to thread 5300
I20260929 14:11:09.078141 5739 raft_consensus.cc:2726] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 14:11:09.078809 5347 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
W20260929 14:11:09.079296 5297 debug-util.cc:398] Leaking SignalData structure 0x7b080027b080 after lost signal to thread 5375
I20260929 14:11:09.080125 5239 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:09.082724 5729 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:09.083698 5729 raft_consensus.cc:686] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:09.084501 5729 consensus_queue.cc:237] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.092672 5391 tablet_bootstrap.cc:654] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:09.104667 5271 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:09.051275 5741 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.114194 5195 raft_consensus.cc:2379] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 1: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:09.116636 5741 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:09.122925 5199 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
W20260929 14:11:09.135442 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 151111us
I20260929 14:11:09.136636 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:09.137534 5345 raft_consensus.cc:2379] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 1: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:09.139142 5393 tablet_bootstrap.cc:492] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:09.139612 5393 ts_tablet_manager.cc:1397] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.222s user 0.024s sys 0.016s
I20260929 14:11:09.139366 5730 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.144609 5344 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:09.147531 5730 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:09.151926 5393 raft_consensus.cc:348] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.152634 5393 raft_consensus.cc:374] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:09.152863 5393 raft_consensus.cc:729] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
W20260929 14:11:09.203794 5147 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260929 14:11:09.207911 5200 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:09.194967 5164 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:09.215054 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:09.217738 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:09.231384 5392 tablet_bootstrap.cc:492] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:09.231319 5316 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:09.232632 5392 ts_tablet_manager.cc:1397] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.297s user 0.007s sys 0.019s
I20260929 14:11:09.237736 5391 tablet_bootstrap.cc:492] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:09.238725 5391 ts_tablet_manager.cc:1397] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.368s user 0.012s sys 0.005s
I20260929 14:11:09.241248 5568 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:09.247802 5739 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:09.254190 5568 raft_consensus.cc:686] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
W20260929 14:11:09.255757 5314 leader_election.cc:341] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:09.257791 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:09.255786 5568 consensus_queue.cc:237] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.248530 5739 raft_consensus.cc:686] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:09.275982 5739 consensus_queue.cc:237] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.285866 5392 raft_consensus.cc:348] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.286813 5392 raft_consensus.cc:374] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:09.287175 5392 raft_consensus.cc:729] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:09.292663 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:09.293696 5195 raft_consensus.cc:3037] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:09.292552 5392 consensus_queue.cc:260] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.274085 5391 raft_consensus.cc:348] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.303838 5391 raft_consensus.cc:374] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:09.304186 5391 raft_consensus.cc:729] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:09.304910 5391 consensus_queue.cc:260] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.307560 5391 ts_tablet_manager.cc:1428] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.068s user 0.005s sys 0.004s
I20260929 14:11:09.315145 5316 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:09.316751 5567 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:09.317615 5392 ts_tablet_manager.cc:1428] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.084s user 0.008s sys 0.002s
I20260929 14:11:09.300153 5393 consensus_queue.cc:260] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.322542 5393 ts_tablet_manager.cc:1428] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.183s user 0.007s sys 0.002s
I20260929 14:11:09.324820 5391 tablet_bootstrap.cc:492] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:09.325138 5098 catalog_manager.cc:5495] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
W20260929 14:11:09.348469 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 102332us
I20260929 14:11:09.351282 5393 tablet_bootstrap.cc:492] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:09.357885 5392 tablet_bootstrap.cc:492] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:09.363932 5195 raft_consensus.cc:2454] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:09.367833 5165 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:09.368948 5793 raft_consensus.cc:2726] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 14:11:09.381636 5740 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:09.382284 5740 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.384935 5740 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:09.385769 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:09.387984 5164 leader_election.cc:341] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:09.405256 5567 consensus_queue.cc:1035] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:09.407817 5391 tablet_bootstrap.cc:654] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:09.417889 5316 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:09.430491 5098 catalog_manager.cc:5495] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:09.435623 5567 raft_consensus.cc:2781] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:09.442503 5267 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:09.448527 5392 tablet_bootstrap.cc:654] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:09.448933 5393 tablet_bootstrap.cc:654] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:09.460227 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:09.450953 5567 raft_consensus.cc:686] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:09.469101 5567 consensus_queue.cc:237] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.476493 5568 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:09.477098 5568 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.478528 5391 tablet_bootstrap.cc:492] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:09.478955 5391 ts_tablet_manager.cc:1397] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.155s user 0.010s sys 0.010s
I20260929 14:11:09.478377 5730 consensus_queue.cc:1035] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260929 14:11:09.482937 5568 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:09.483163 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:09.483999 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:09.485342 5314 leader_election.cc:341] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:09.499809 5391 raft_consensus.cc:348] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.502900 5391 raft_consensus.cc:374] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:09.509011 5391 raft_consensus.cc:729] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:09.511168 5391 consensus_queue.cc:260] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.514571 5199 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:09.530308 5316 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:09.531711 5567 raft_consensus.cc:2781] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:09.532080 5567 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:09.532416 5567 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:11:09.560182 5165 leader_election.cc:341] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:09.560640 5165 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:09.562456 5793 raft_consensus.cc:2726] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:09.570762 5567 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.572656 5567 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:09.573858 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:09.574494 5199 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:09.580904 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:09.581923 5199 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:09.585183 5393 tablet_bootstrap.cc:492] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:09.586225 5316 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:09.592553 5314 leader_election.cc:341] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:09.595482 5393 ts_tablet_manager.cc:1397] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.245s user 0.019s sys 0.005s
I20260929 14:11:09.595858 5567 raft_consensus.cc:2781] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:09.598232 5391 ts_tablet_manager.cc:1428] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.119s user 0.013s sys 0.006s
I20260929 14:11:09.603726 5391 tablet_bootstrap.cc:492] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:09.606086 5393 raft_consensus.cc:348] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.606945 5393 raft_consensus.cc:374] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:09.607267 5393 raft_consensus.cc:729] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:09.601161 5098 catalog_manager.cc:5495] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:09.602408 5567 raft_consensus.cc:686] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:09.608078 5393 consensus_queue.cc:260] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.608561 5567 consensus_queue.cc:237] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.610671 5393 ts_tablet_manager.cc:1428] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260929 14:11:09.610844 5392 tablet_bootstrap.cc:492] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:09.611322 5392 ts_tablet_manager.cc:1397] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.254s user 0.016s sys 0.008s
I20260929 14:11:09.611989 5393 tablet_bootstrap.cc:492] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:09.623905 5391 tablet_bootstrap.cc:654] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:09.626174 5727 consensus_queue.cc:1035] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:09.630249 5392 raft_consensus.cc:348] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.635663 5392 raft_consensus.cc:374] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:09.635970 5392 raft_consensus.cc:729] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:09.636807 5392 consensus_queue.cc:260] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.641399 5392 ts_tablet_manager.cc:1428] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.030s user 0.005s sys 0.004s
I20260929 14:11:09.642390 5392 tablet_bootstrap.cc:492] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:09.646621 5393 tablet_bootstrap.cc:654] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:09.658668 5665 consensus_queue.cc:1035] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:09.660199 5392 tablet_bootstrap.cc:654] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:09.679522 5391 tablet_bootstrap.cc:492] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:09.680056 5391 ts_tablet_manager.cc:1397] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.077s user 0.012s sys 0.002s
I20260929 14:11:09.682943 5391 raft_consensus.cc:348] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.683738 5391 raft_consensus.cc:374] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:09.684043 5391 raft_consensus.cc:729] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:09.684680 5391 consensus_queue.cc:260] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.697623 5391 ts_tablet_manager.cc:1428] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.017s user 0.007s sys 0.001s
I20260929 14:11:09.698740 5391 tablet_bootstrap.cc:492] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd: Bootstrap starting.
I20260929 14:11:09.707639 5393 tablet_bootstrap.cc:492] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:09.708089 5393 ts_tablet_manager.cc:1397] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.096s user 0.017s sys 0.000s
I20260929 14:11:09.711237 5393 raft_consensus.cc:348] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.712131 5393 raft_consensus.cc:374] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:09.712428 5393 raft_consensus.cc:729] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:09.713297 5393 consensus_queue.cc:260] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.715824 5391 tablet_bootstrap.cc:654] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:09.721496 5392 tablet_bootstrap.cc:492] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:09.722038 5392 ts_tablet_manager.cc:1397] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.080s user 0.016s sys 0.000s
I20260929 14:11:09.725538 5098 catalog_manager.cc:5495] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:09.733292 5098 catalog_manager.cc:5495] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:09.734018 5392 raft_consensus.cc:348] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.738929 5392 raft_consensus.cc:374] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:09.739228 5392 raft_consensus.cc:729] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:09.734670 5391 tablet_bootstrap.cc:492] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd: No bootstrap required, opened a new log
I20260929 14:11:09.735641 5393 ts_tablet_manager.cc:1428] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20260929 14:11:09.739914 5391 ts_tablet_manager.cc:1397] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd: Time spent bootstrapping tablet: real 0.041s user 0.011s sys 0.000s
I20260929 14:11:09.741477 5393 tablet_bootstrap.cc:492] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:09.741997 5392 consensus_queue.cc:260] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.742763 5391 raft_consensus.cc:348] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.743423 5391 raft_consensus.cc:374] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:09.743669 5391 raft_consensus.cc:729] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Initialized, Role: FOLLOWER
I20260929 14:11:09.744186 5391 consensus_queue.cc:260] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.773685 5391 ts_tablet_manager.cc:1428] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20260929 14:11:09.782056 5392 ts_tablet_manager.cc:1428] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.060s user 0.006s sys 0.000s
I20260929 14:11:09.783155 5784 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:09.783620 5784 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.785554 5784 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:09.787382 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:09.788026 5267 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:09.790258 5314 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:09.795228 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:09.795925 5200 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:09.802479 5392 tablet_bootstrap.cc:492] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:09.804638 5762 consensus_queue.cc:1035] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260929 14:11:09.800844 5784 raft_consensus.cc:2781] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:09.815660 5784 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:09.816184 5784 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:09.820690 5393 tablet_bootstrap.cc:654] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:09.820999 5567 consensus_queue.cc:1035] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:09.825179 5392 tablet_bootstrap.cc:654] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
W20260929 14:11:09.838003 5165 consensus_peers.cc:487] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd -> Peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Couldn't send request to peer c5ec116e39144881bdf1f2cd3e923cf5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:09.851076 5393 tablet_bootstrap.cc:492] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:09.851686 5393 ts_tablet_manager.cc:1397] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.110s user 0.011s sys 0.011s
I20260929 14:11:09.854168 5393 raft_consensus.cc:348] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.854928 5393 raft_consensus.cc:374] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:09.855223 5393 raft_consensus.cc:729] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
W20260929 14:11:09.856212 5314 consensus_peers.cc:487] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 -> Peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Couldn't send request to peer c5ec116e39144881bdf1f2cd3e923cf5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:09.856777 5753 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:09.857263 5753 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.859282 5753 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:09.860630 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:09.862357 5165 leader_election.cc:341] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:09.862151 5784 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.864012 5784 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:09.864742 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:09.865345 5199 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:09.867694 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:09.869403 5392 tablet_bootstrap.cc:492] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:09.870659 5392 ts_tablet_manager.cc:1397] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.068s user 0.012s sys 0.004s
I20260929 14:11:09.873842 5345 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:09.873919 5392 raft_consensus.cc:348] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.875690 5392 raft_consensus.cc:374] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:09.875989 5392 raft_consensus.cc:729] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:09.876629 5392 consensus_queue.cc:260] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.878219 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:09.881750 5267 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:09.887250 5164 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:09.889338 5762 raft_consensus.cc:2781] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:09.889708 5762 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:09.889999 5762 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:09.892973 5199 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:09.894714 5267 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:09.897607 5392 ts_tablet_manager.cc:1428] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.027s user 0.005s sys 0.001s
I20260929 14:11:09.898159 5316 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:09.899048 5759 raft_consensus.cc:2781] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:09.899472 5759 raft_consensus.cc:686] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:09.900220 5759 consensus_queue.cc:237] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.905920 5392 tablet_bootstrap.cc:492] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:09.855980 5393 consensus_queue.cc:260] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.913985 5393 ts_tablet_manager.cc:1428] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.062s user 0.005s sys 0.000s
I20260929 14:11:09.914885 5392 tablet_bootstrap.cc:654] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:09.906757 5762 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.923143 5762 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:09.923976 5393 tablet_bootstrap.cc:492] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:09.924083 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:11:09.928556 5165 leader_election.cc:341] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:09.935012 5393 tablet_bootstrap.cc:654] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
W20260929 14:11:09.936118 5314 consensus_peers.cc:487] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 -> Peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Couldn't send request to peer c5ec116e39144881bdf1f2cd3e923cf5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:09.941654 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:09.942804 5346 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:09.952934 5346 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:09.952823 5098 catalog_manager.cc:5495] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:09.954944 5392 tablet_bootstrap.cc:492] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:09.955056 5164 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:09.955489 5392 ts_tablet_manager.cc:1397] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.050s user 0.013s sys 0.003s
I20260929 14:11:09.958293 5392 raft_consensus.cc:348] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.959137 5739 raft_consensus.cc:2781] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:09.959645 5739 raft_consensus.cc:686] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:09.960418 5739 consensus_queue.cc:237] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.964376 5392 raft_consensus.cc:374] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:09.964713 5392 raft_consensus.cc:729] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:09.965374 5392 consensus_queue.cc:260] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:09.967501 5392 ts_tablet_manager.cc:1428] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260929 14:11:09.972887 5392 tablet_bootstrap.cc:492] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:09.986083 5096 catalog_manager.cc:5495] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:10.014828 5392 tablet_bootstrap.cc:654] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:10.015182 5267 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:10.018462 5393 tablet_bootstrap.cc:492] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:10.022881 5393 ts_tablet_manager.cc:1397] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.099s user 0.019s sys 0.004s
I20260929 14:11:10.028007 5393 raft_consensus.cc:348] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.029062 5393 raft_consensus.cc:374] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:10.029475 5393 raft_consensus.cc:729] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:10.030347 5393 consensus_queue.cc:260] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.042120 5759 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:10.042632 5759 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.044131 5393 ts_tablet_manager.cc:1428] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.021s user 0.005s sys 0.001s
I20260929 14:11:10.044590 5759 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:10.048545 5393 tablet_bootstrap.cc:492] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4: Bootstrap starting.
I20260929 14:11:10.046746 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:10.050622 5314 leader_election.cc:341] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:10.054179 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:10.054924 5196 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:10.056183 5316 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:10.058022 5567 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:10.058367 5567 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:10.058677 5567 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:10.062915 5727 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:10.063385 5727 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.063977 5392 tablet_bootstrap.cc:492] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:10.064433 5392 ts_tablet_manager.cc:1397] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.092s user 0.009s sys 0.011s
I20260929 14:11:10.065778 5727 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:10.067111 5392 raft_consensus.cc:348] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.067826 5392 raft_consensus.cc:374] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:10.068063 5392 raft_consensus.cc:729] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:10.068404 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:10.069187 5346 raft_consensus.cc:2379] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 1: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:10.068650 5392 consensus_queue.cc:260] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.062341 5393 tablet_bootstrap.cc:654] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:10.076424 5392 ts_tablet_manager.cc:1428] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260929 14:11:10.077508 5392 tablet_bootstrap.cc:492] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:10.078002 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:10.078933 5199 raft_consensus.cc:2379] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 1: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:10.081681 5242 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:10.082774 5465 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:10.091439 5392 tablet_bootstrap.cc:654] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:10.091502 5465 raft_consensus.cc:2726] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:10.096788 5393 tablet_bootstrap.cc:492] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4: No bootstrap required, opened a new log
I20260929 14:11:10.097414 5393 ts_tablet_manager.cc:1397] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.006s
I20260929 14:11:10.100224 5393 raft_consensus.cc:348] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.101063 5393 raft_consensus.cc:374] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:10.100852 5759 consensus_queue.cc:1035] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:10.101437 5393 raft_consensus.cc:729] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Initialized, Role: FOLLOWER
I20260929 14:11:10.102545 5393 consensus_queue.cc:260] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.105147 5393 ts_tablet_manager.cc:1428] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 14:11:10.123292 5392 tablet_bootstrap.cc:492] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:10.123775 5392 ts_tablet_manager.cc:1397] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.004s
I20260929 14:11:10.126483 5392 raft_consensus.cc:348] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.127179 5392 raft_consensus.cc:374] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:10.127434 5392 raft_consensus.cc:729] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:10.128062 5392 consensus_queue.cc:260] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.130381 5392 ts_tablet_manager.cc:1428] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260929 14:11:10.131824 5392 tablet_bootstrap.cc:492] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:10.136651 5392 tablet_bootstrap.cc:654] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:10.139022 5465 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:10.136837 5567 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.139699 5465 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.141960 5465 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:10.142547 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:10.143857 5199 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:10.143721 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:10.151863 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:11:10.153331 5314 leader_election.cc:341] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:10.154649 5567 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:10.154569 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:10.157044 5242 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:10.160827 5465 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:10.163579 5199 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:10.164922 5316 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:10.166373 5759 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:10.167128 5759 raft_consensus.cc:686] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:10.167974 5759 consensus_queue.cc:237] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.198588 5727 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:10.199132 5727 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.206487 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:10.214579 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:10.205082 5727 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:10.218418 5567 consensus_queue.cc:1035] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:10.220497 5242 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:10.234266 5465 raft_consensus.cc:2726] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:10.267724 5098 catalog_manager.cc:5495] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:10.269901 5392 tablet_bootstrap.cc:492] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:10.270601 5392 ts_tablet_manager.cc:1397] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.139s user 0.015s sys 0.027s
I20260929 14:11:10.277977 5762 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:10.278776 5762 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.285925 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:10.287829 5165 leader_election.cc:341] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:10.298342 5794 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:10.304756 5762 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:10.322324 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:10.323105 5345 raft_consensus.cc:2454] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 0.
I20260929 14:11:10.324286 5164 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:10.325054 5762 raft_consensus.cc:2781] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:10.325449 5762 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:10.325829 5762 raft_consensus.cc:3037] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:10.305588 5794 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.328115 5794 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:10.331486 5392 raft_consensus.cc:348] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.332234 5392 raft_consensus.cc:374] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:10.332515 5392 raft_consensus.cc:729] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:10.333130 5762 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.333328 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:10.340338 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:10.333144 5392 consensus_queue.cc:260] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.347605 5392 ts_tablet_manager.cc:1428] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.077s user 0.004s sys 0.003s
I20260929 14:11:10.352824 5239 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:10.353564 5762 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:10.354262 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:10.354848 5392 tablet_bootstrap.cc:492] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:10.355175 5665 raft_consensus.cc:2726] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260929 14:11:10.357355 5165 leader_election.cc:341] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:10.357452 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:10.358577 5345 raft_consensus.cc:3037] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:11:10.374105 5314 consensus_peers.cc:487] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 -> Peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Couldn't send request to peer c5ec116e39144881bdf1f2cd3e923cf5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:10.376739 5275 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:10.384896 5794 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:10.385551 5794 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.386929 5345 raft_consensus.cc:2454] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:10.387495 5794 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:10.388116 5164 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:10.388779 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:10.389832 5350 raft_consensus.cc:2379] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 1: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:10.390292 5762 raft_consensus.cc:2781] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:10.390811 5762 raft_consensus.cc:686] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:10.391953 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:10.372998 5784 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:10.393741 5784 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.395650 5784 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:10.392791 5762 consensus_queue.cc:237] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.397831 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:10.399823 5200 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 0.
I20260929 14:11:10.400312 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:10.398320 5239 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:10.402024 5465 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
W20260929 14:11:10.402506 5314 leader_election.cc:341] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:10.404258 5392 tablet_bootstrap.cc:654] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:10.407177 5743 consensus_queue.cc:1035] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:10.420249 5784 consensus_queue.cc:1035] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:10.421451 5316 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:10.423708 5759 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 14:11:10.424077 5759 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:10.424393 5759 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:10.433450 5465 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:10.439051 5269 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:10.448071 5759 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.453176 5741 consensus_queue.cc:1035] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260929 14:11:10.454370 5147 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260929 14:11:10.464563 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:10.465252 5199 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:10.465130 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:11:10.466528 5314 leader_election.cc:341] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Illegal state: must be running to vote when last-logged opid is not known
I20260929 14:11:10.470314 5759 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:10.472724 5199 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:10.474313 5316 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:10.481559 5759 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 14:11:10.483855 5759 raft_consensus.cc:686] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:10.501960 5784 consensus_queue.cc:1035] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 14:11:10.506125 5392 tablet_bootstrap.cc:492] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:10.506691 5392 ts_tablet_manager.cc:1397] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.152s user 0.007s sys 0.007s
I20260929 14:11:10.501961 5759 consensus_queue.cc:237] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.518904 5392 raft_consensus.cc:348] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.519779 5392 raft_consensus.cc:374] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:10.520066 5392 raft_consensus.cc:729] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:10.520726 5392 consensus_queue.cc:260] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.523097 5392 ts_tablet_manager.cc:1428] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.016s user 0.005s sys 0.000s
I20260929 14:11:10.530345 5099 catalog_manager.cc:5495] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 1, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:10.541812 5269 raft_consensus.cc:3037] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:10.554597 5099 catalog_manager.cc:5495] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:10.559329 5784 consensus_queue.cc:1035] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:10.567565 5392 tablet_bootstrap.cc:492] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:10.584125 5392 tablet_bootstrap.cc:654] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
W20260929 14:11:10.593290 5314 consensus_peers.cc:487] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 -> Peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Couldn't send request to peer c5ec116e39144881bdf1f2cd3e923cf5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:10.615156 5392 tablet_bootstrap.cc:492] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:10.615645 5392 ts_tablet_manager.cc:1397] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.048s user 0.012s sys 0.004s
I20260929 14:11:10.618170 5392 raft_consensus.cc:348] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.619405 5392 raft_consensus.cc:374] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:10.619689 5392 raft_consensus.cc:729] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:10.621899 5392 consensus_queue.cc:260] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.624776 5392 ts_tablet_manager.cc:1428] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.009s user 0.007s sys 0.002s
I20260929 14:11:10.626057 5392 tablet_bootstrap.cc:492] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:10.632210 5392 tablet_bootstrap.cc:654] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:10.663332 5392 tablet_bootstrap.cc:492] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:10.663839 5392 ts_tablet_manager.cc:1397] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.006s
I20260929 14:11:10.667111 5392 raft_consensus.cc:348] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.667815 5392 raft_consensus.cc:374] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:10.668076 5392 raft_consensus.cc:729] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:10.668788 5392 consensus_queue.cc:260] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.674352 5741 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:10.674820 5741 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.677554 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:10.677771 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:10.678061 5350 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:10.678401 5269 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:10.679432 5165 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:10.679473 5741 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:10.679953 5392 ts_tablet_manager.cc:1428] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.016s user 0.005s sys 0.001s
I20260929 14:11:10.680249 5741 raft_consensus.cc:2781] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:10.680603 5741 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:10.680948 5741 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:10.689893 5741 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.680938 5392 tablet_bootstrap.cc:492] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5: Bootstrap starting.
I20260929 14:11:10.704276 5392 tablet_bootstrap.cc:654] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5: Neither blocks nor log segments found. Creating new log.
I20260929 14:11:10.705855 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:10.706502 5269 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:10.706609 5741 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:10.708668 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:10.709288 5350 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:10.715986 5350 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:10.716966 5164 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:10.717686 5741 raft_consensus.cc:2781] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:10.718200 5741 raft_consensus.cc:686] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:10.719038 5741 consensus_queue.cc:237] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.717701 5269 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:10.731647 5392 tablet_bootstrap.cc:492] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5: No bootstrap required, opened a new log
I20260929 14:11:10.733402 5392 ts_tablet_manager.cc:1397] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.018s
I20260929 14:11:10.737455 5392 raft_consensus.cc:348] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.738212 5392 raft_consensus.cc:374] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 14:11:10.738519 5392 raft_consensus.cc:729] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Initialized, Role: FOLLOWER
I20260929 14:11:10.739164 5392 consensus_queue.cc:260] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.741684 5392 ts_tablet_manager.cc:1428] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260929 14:11:10.757140 5097 catalog_manager.cc:5495] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 0 to 2, leader changed from <none> to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 2 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:10.759733 5794 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:10.760324 5794 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.762358 5794 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:10.763965 5269 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:10.770723 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:10.776087 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:10.777734 5242 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:10.787474 5794 raft_consensus.cc:2726] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:10.900156 5794 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:10.900676 5794 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:10.903059 5794 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:10.904664 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:10.905584 5200 raft_consensus.cc:2379] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 1: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:10.908272 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:10.909853 5239 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:10.910853 5727 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:10.928668 5269 raft_consensus.cc:3037] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:10.933128 5727 raft_consensus.cc:2726] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:10.944085 5741 consensus_queue.cc:1035] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:10.966837 5743 consensus_queue.cc:1035] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:11.029305 5759 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:11.050762 5759 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:11.112623 5727 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:11.113130 5727 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:11.115186 5727 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:11.116796 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:11.126161 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:11.130008 5242 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:11.131315 5727 raft_consensus.cc:2726] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 14:11:11.139868 5346 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260929 14:11:11.149866 5270 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 14:11:11.154610 5753 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 14:11:11.173283 5269 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260929 14:11:11.178251 5762 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
W20260929 14:11:11.643635 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 59915.6us
I20260929 14:11:12.323068 4041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 14:11:12.351670 5860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 14:11:12.362522 5861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:11:12.378669 4041 server_base.cc:1034] running on GCE node
W20260929 14:11:12.379258 5863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 14:11:12.380582 4041 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 14:11:12.380918 4041 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 14:11:12.381091 4041 hybrid_clock.cc:648] HybridClock initialized: now 1790691072381079 us; error 0 us; skew 500 ppm
I20260929 14:11:12.381732 4041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 14:11:12.391139 4041 webserver.cc:460] Webserver started at http://127.3.242.68:36303/ using document root <none> and password file <none>
I20260929 14:11:12.391899 4041 fs_manager.cc:359] Metadata directory not provided
I20260929 14:11:12.392180 4041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 14:11:12.392570 4041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 14:11:12.394224 4041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-3-root/instance:
uuid: "675b77a0a88a4bf08134281fc0cd949e"
format_stamp: "Formatted at 2026-09-29 14:11:12 on dist-test-slave-6gv9"
I20260929 14:11:12.408833 4041 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.003s sys 0.007s
I20260929 14:11:12.455878 5869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 14:11:12.456889 4041 fs_manager.cc:725] Time spent opening block manager: real 0.042s user 0.004s sys 0.007s
I20260929 14:11:12.457284 4041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-3-root
uuid: "675b77a0a88a4bf08134281fc0cd949e"
format_stamp: "Formatted at 2026-09-29 14:11:12 on dist-test-slave-6gv9"
I20260929 14:11:12.457656 4041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 14:11:12.572610 4041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 14:11:12.574666 4041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 14:11:12.589732 4041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 14:11:12.599973 4041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 14:11:12.600364 4041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 14:11:12.600813 4041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 14:11:12.601101 4041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 14:11:12.957352 4041 rpc_server.cc:307] RPC server started. Bound to: 127.3.242.68:35497
I20260929 14:11:12.957610 5932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.242.68:35497 every 8 connection(s)
I20260929 14:11:12.976300 4041 auto_leader_rebalancer-test.cc:413] add a tserver: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:13.277384 5933 heartbeater.cc:347] Connected to a master server at 127.3.242.126:41327
I20260929 14:11:13.277938 5933 heartbeater.cc:464] Registering TS with master...
I20260929 14:11:13.278832 5933 heartbeater.cc:511] Master 127.3.242.126:41327 requested a full tablet report, sending...
I20260929 14:11:13.283779 5096 ts_manager.cc:194] Registered new tserver with Master: 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:13.296010 5096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:44250
I20260929 14:11:13.963434 5200 consensus_queue.cc:237] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:13.980211 5346 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:13.980713 5270 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:13.982110 5762 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:14.010776 5762 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:14.024544 5741 raft_consensus.cc:2932] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:14.026492 5270 raft_consensus.cc:2932] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
W20260929 14:11:14.031201 5167 consensus_peers.cc:487] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5acb0315ba44b5a9d145119868e7168. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:14.057159 5345 raft_consensus.cc:2932] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
W20260929 14:11:14.155778 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 94865.6us
I20260929 14:11:14.176405 5096 catalog_manager.cc:5495] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:14.228609 5345 consensus_queue.cc:237] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
W20260929 14:11:14.265856 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 92771.4us
W20260929 14:11:14.318202 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 120468us
I20260929 14:11:14.327858 5267 raft_consensus.cc:1261] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:14.337641 5200 raft_consensus.cc:1261] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:14.339702 5855 consensus_queue.cc:1035] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260929 14:11:14.346873 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 50204.9us
I20260929 14:11:14.349802 5567 consensus_queue.cc:1035] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:14.358791 5933 heartbeater.cc:503] Master 127.3.242.126:41327 was elected leader, sending a full tablet report...
W20260929 14:11:14.404371 5316 consensus_peers.cc:487] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4040985909414481ad0d1491249e8e67. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:14.406317 5960 ts_tablet_manager.cc:927] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147)
I20260929 14:11:14.410502 5730 raft_consensus.cc:2932] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:14.412568 5200 raft_consensus.cc:2932] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:14.444468 5960 tablet_copy_client.cc:316] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.65:45147
I20260929 14:11:14.450114 5267 raft_consensus.cc:2932] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:14.586405 5097 catalog_manager.cc:5495] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:14.613412 5267 consensus_queue.cc:237] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:14.617383 5211 tablet_copy_service.cc:140] P ac17d3b553b74c9da39e0ab307addacd: Received BeginTabletCopySession request for tablet e5acb0315ba44b5a9d145119868e7168 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:37270)
I20260929 14:11:14.618052 5211 tablet_copy_service.cc:161] P ac17d3b553b74c9da39e0ab307addacd: Beginning new tablet copy session on tablet e5acb0315ba44b5a9d145119868e7168 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:37270: session id = 675b77a0a88a4bf08134281fc0cd949e-e5acb0315ba44b5a9d145119868e7168
I20260929 14:11:14.629760 5344 raft_consensus.cc:1261] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:14.643296 5211 tablet_copy_source_session.cc:205] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:14.648468 5957 consensus_queue.cc:1035] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:14.651299 5197 raft_consensus.cc:1261] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:14.656243 5957 consensus_queue.cc:1035] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:14.658907 5960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5acb0315ba44b5a9d145119868e7168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:14.689643 5949 raft_consensus.cc:2932] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:14.699011 5197 raft_consensus.cc:2932] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:14.712007 5097 catalog_manager.cc:5495] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:14.719815 5267 consensus_queue.cc:237] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:14.723639 5960 tablet_copy_client.cc:791] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:14.727401 5960 tablet_copy_client.cc:655] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
W20260929 14:11:14.737038 5242 consensus_peers.cc:487] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6978e56e8f49441795cc7f6b6a8f4cfe. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:14.741503 5345 raft_consensus.cc:2932] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:14.751888 5960 tablet_copy_client.cc:531] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:14.780700 5960 tablet_bootstrap.cc:492] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
W20260929 14:11:14.832070 5242 consensus_peers.cc:487] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5e230699b3fb4d4d97fdb04988babce5. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:14.841665 5345 raft_consensus.cc:1261] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:14.848970 5957 consensus_queue.cc:1035] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:14.842578 5972 ts_tablet_manager.cc:927] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:14.859758 5197 raft_consensus.cc:1261] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:14.861547 5957 consensus_queue.cc:1035] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:14.873515 5972 tablet_copy_client.cc:316] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.67:42639
I20260929 14:11:14.881474 5960 tablet_bootstrap.cc:492] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:14.882499 5960 tablet_bootstrap.cc:492] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:14.883176 5960 ts_tablet_manager.cc:1397] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.103s user 0.019s sys 0.011s
I20260929 14:11:14.885177 5197 raft_consensus.cc:2932] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:14.883512 5957 raft_consensus.cc:2932] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:14.898823 5960 raft_consensus.cc:348] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:14.900151 5960 raft_consensus.cc:729] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:14.901019 5960 consensus_queue.cc:260] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:14.925551 5345 raft_consensus.cc:2932] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:14.931964 5361 tablet_copy_service.cc:140] P d886fb1eb0474d2d9f7410c4013c45e4: Received BeginTabletCopySession request for tablet 4040985909414481ad0d1491249e8e67 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:38536)
I20260929 14:11:14.932799 5361 tablet_copy_service.cc:161] P d886fb1eb0474d2d9f7410c4013c45e4: Beginning new tablet copy session on tablet 4040985909414481ad0d1491249e8e67 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:38536: session id = 675b77a0a88a4bf08134281fc0cd949e-4040985909414481ad0d1491249e8e67
I20260929 14:11:14.942619 5960 ts_tablet_manager.cc:1428] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.059s user 0.007s sys 0.006s
I20260929 14:11:14.946821 5211 tablet_copy_service.cc:342] P ac17d3b553b74c9da39e0ab307addacd: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-e5acb0315ba44b5a9d145119868e7168 received from {username='slave'} at 127.0.0.1:37270
I20260929 14:11:14.947304 5211 tablet_copy_service.cc:433] P ac17d3b553b74c9da39e0ab307addacd: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-e5acb0315ba44b5a9d145119868e7168 on tablet e5acb0315ba44b5a9d145119868e7168 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:14.937704 5097 catalog_manager.cc:5495] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:14.968319 5361 tablet_copy_source_session.cc:205] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:14.989454 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4040985909414481ad0d1491249e8e67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:15.031744 5972 tablet_copy_client.cc:791] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:15.032473 5972 tablet_copy_client.cc:655] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:15.040076 5972 tablet_copy_client.cc:531] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:15.055871 5972 tablet_bootstrap.cc:492] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:15.139496 5972 tablet_bootstrap.cc:492] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:15.140473 5972 tablet_bootstrap.cc:492] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:15.141178 5972 ts_tablet_manager.cc:1397] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.086s user 0.024s sys 0.006s
I20260929 14:11:15.147563 5972 raft_consensus.cc:348] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:15.148545 5972 raft_consensus.cc:729] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:15.149537 5972 consensus_queue.cc:260] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:15.155210 5972 ts_tablet_manager.cc:1428] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20260929 14:11:15.165521 5361 tablet_copy_service.cc:342] P d886fb1eb0474d2d9f7410c4013c45e4: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-4040985909414481ad0d1491249e8e67 received from {username='slave'} at 127.0.0.1:38536
I20260929 14:11:15.165951 5361 tablet_copy_service.cc:433] P d886fb1eb0474d2d9f7410c4013c45e4: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-4040985909414481ad0d1491249e8e67 on tablet 4040985909414481ad0d1491249e8e67 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:15.252441 5972 ts_tablet_manager.cc:927] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:15.279269 5972 tablet_copy_client.cc:316] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.66:43473
I20260929 14:11:15.293855 5907 raft_consensus.cc:1203] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:15.313653 5286 tablet_copy_service.cc:140] P c5ec116e39144881bdf1f2cd3e923cf5: Received BeginTabletCopySession request for tablet 6978e56e8f49441795cc7f6b6a8f4cfe from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:41626)
I20260929 14:11:15.314278 5286 tablet_copy_service.cc:161] P c5ec116e39144881bdf1f2cd3e923cf5: Beginning new tablet copy session on tablet 6978e56e8f49441795cc7f6b6a8f4cfe from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:41626: session id = 675b77a0a88a4bf08134281fc0cd949e-6978e56e8f49441795cc7f6b6a8f4cfe
I20260929 14:11:15.329561 5960 ts_tablet_manager.cc:927] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:15.332165 5286 tablet_copy_source_session.cc:205] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:15.335769 5972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6978e56e8f49441795cc7f6b6a8f4cfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:15.347056 5907 raft_consensus.cc:1203] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:15.371273 5972 tablet_copy_client.cc:791] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:15.371974 5972 tablet_copy_client.cc:655] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:15.390427 5972 tablet_copy_client.cc:531] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:15.418290 5972 tablet_bootstrap.cc:492] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:15.507194 5972 tablet_bootstrap.cc:492] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:15.508311 5972 tablet_bootstrap.cc:492] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:15.509076 5972 ts_tablet_manager.cc:1397] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.091s user 0.022s sys 0.009s
I20260929 14:11:15.512315 5972 raft_consensus.cc:348] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:15.514515 5972 raft_consensus.cc:729] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:15.516417 5972 consensus_queue.cc:260] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:15.531368 5972 ts_tablet_manager.cc:1428] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.022s user 0.000s sys 0.006s
I20260929 14:11:15.542526 5286 tablet_copy_service.cc:342] P c5ec116e39144881bdf1f2cd3e923cf5: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-6978e56e8f49441795cc7f6b6a8f4cfe received from {username='slave'} at 127.0.0.1:41626
I20260929 14:11:15.542987 5286 tablet_copy_service.cc:433] P c5ec116e39144881bdf1f2cd3e923cf5: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-6978e56e8f49441795cc7f6b6a8f4cfe on tablet 6978e56e8f49441795cc7f6b6a8f4cfe with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:15.553728 5960 tablet_copy_client.cc:316] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.66:43473
I20260929 14:11:15.555641 5286 tablet_copy_service.cc:140] P c5ec116e39144881bdf1f2cd3e923cf5: Received BeginTabletCopySession request for tablet 5e230699b3fb4d4d97fdb04988babce5 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:41626)
I20260929 14:11:15.556200 5286 tablet_copy_service.cc:161] P c5ec116e39144881bdf1f2cd3e923cf5: Beginning new tablet copy session on tablet 5e230699b3fb4d4d97fdb04988babce5 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:41626: session id = 675b77a0a88a4bf08134281fc0cd949e-5e230699b3fb4d4d97fdb04988babce5
I20260929 14:11:15.564992 5286 tablet_copy_source_session.cc:205] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:15.571506 5960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e230699b3fb4d4d97fdb04988babce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:15.609448 5960 tablet_copy_client.cc:791] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:15.610190 5960 tablet_copy_client.cc:655] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:15.614076 5960 tablet_copy_client.cc:531] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:15.626924 5960 tablet_bootstrap.cc:492] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:15.695567 5960 tablet_bootstrap.cc:492] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:15.696697 5960 tablet_bootstrap.cc:492] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:15.703822 5960 ts_tablet_manager.cc:1397] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.077s user 0.028s sys 0.010s
I20260929 14:11:15.706831 5960 raft_consensus.cc:348] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:15.707978 5960 raft_consensus.cc:729] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:15.709134 5960 consensus_queue.cc:260] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:15.713140 5960 ts_tablet_manager.cc:1428] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260929 14:11:15.716156 5286 tablet_copy_service.cc:342] P c5ec116e39144881bdf1f2cd3e923cf5: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-5e230699b3fb4d4d97fdb04988babce5 received from {username='slave'} at 127.0.0.1:41626
I20260929 14:11:15.716562 5286 tablet_copy_service.cc:433] P c5ec116e39144881bdf1f2cd3e923cf5: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-5e230699b3fb4d4d97fdb04988babce5 on tablet 5e230699b3fb4d4d97fdb04988babce5 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:15.723519 5907 raft_consensus.cc:1203] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:15.744989 5947 raft_consensus.cc:1050] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:15.747928 5947 consensus_queue.cc:237] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:15.760700 5855 raft_consensus.cc:1050] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:15.770907 5907 raft_consensus.cc:1203] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:15.809134 5855 consensus_queue.cc:237] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:15.912227 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 108119us
W20260929 14:11:15.914366 5987 log.cc:924] Time spent T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.075s user 0.002s sys 0.000s
I20260929 14:11:15.938774 5907 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:15.943025 5953 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:15.978427 5276 raft_consensus.cc:1261] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:15.985455 5907 raft_consensus.cc:1261] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:15.987932 5730 consensus_queue.cc:1035] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:15.988165 5275 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:15.991231 5730 consensus_queue.cc:1035] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:15.993379 5951 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:16.002061 5345 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:16.004136 5951 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:16.019906 5197 raft_consensus.cc:1261] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:16.046343 5730 consensus_queue.cc:1035] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:16.090236 5947 raft_consensus.cc:2932] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:16.102843 5269 raft_consensus.cc:2932] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:16.115291 5730 raft_consensus.cc:2932] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:16.120587 5345 raft_consensus.cc:2932] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:16.128446 5906 raft_consensus.cc:2932] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:16.130105 5098 catalog_manager.cc:5495] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:16.145783 5197 raft_consensus.cc:2932] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:16.152264 5907 raft_consensus.cc:2932] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:16.179524 5267 raft_consensus.cc:2932] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:16.188529 5096 catalog_manager.cc:5495] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
W20260929 14:11:16.332058 5997 log.cc:924] Time spent T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29: Append to log took a long time: real 0.076s user 0.003s sys 0.000s
I20260929 14:11:16.386901 5197 consensus_queue.cc:237] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:16.407665 5868 raft_consensus.cc:1050] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:16.408677 5949 raft_consensus.cc:1050] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:16.410415 5868 consensus_queue.cc:237] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:16.411228 5949 consensus_queue.cc:237] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:16.421576 5907 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:16.438146 5269 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:16.446897 5741 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:11:16.454703 5951 raft_consensus.cc:2932] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:16.456833 5906 raft_consensus.cc:2932] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:16.465937 5086 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e5acb0315ba44b5a9d145119868e7168 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:16.485999 5346 consensus_queue.cc:237] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:16.494215 5197 raft_consensus.cc:1261] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:16.495924 5985 consensus_queue.cc:1035] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:11:16.495575 5099 catalog_manager.cc:5495] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:16.499523 5907 raft_consensus.cc:1261] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:16.503418 5989 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:11:16.516618 5337 tablet_service.cc:1514] Processing DeleteTablet for tablet e5acb0315ba44b5a9d145119868e7168 with delete_type TABLET_DATA_TOMBSTONED (TS d886fb1eb0474d2d9f7410c4013c45e4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47246
I20260929 14:11:16.522989 5985 consensus_queue.cc:1035] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:11:16.546836 6004 tablet_replica.cc:330] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4: stopping tablet replica
I20260929 14:11:16.548117 6004 raft_consensus.cc:2229] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:16.548972 6004 raft_consensus.cc:2258] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:16.552583 6004 ts_tablet_manager.cc:1905] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:16.572611 6004 ts_tablet_manager.cc:1918] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:16.573236 6004 log.cc:1196] T e5acb0315ba44b5a9d145119868e7168 P d886fb1eb0474d2d9f7410c4013c45e4: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-2-root/wals/e5acb0315ba44b5a9d145119868e7168
I20260929 14:11:16.575105 5083 catalog_manager.cc:4841] TS d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): tablet e5acb0315ba44b5a9d145119868e7168 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:16.587689 5346 raft_consensus.cc:1261] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260929 14:11:16.606980 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 131382us
I20260929 14:11:16.612260 5986 raft_consensus.cc:2932] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:16.615146 5197 raft_consensus.cc:2932] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:16.639963 5099 catalog_manager.cc:5495] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:16.707059 5906 raft_consensus.cc:2932] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:16.714002 5868 consensus_queue.cc:1035] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260929 14:11:16.721647 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 113816us
W20260929 14:11:16.738814 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 64098us
I20260929 14:11:16.754657 5267 raft_consensus.cc:2932] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:16.778270 5083 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4040985909414481ad0d1491249e8e67 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260929 14:11:16.802754 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 59358.8us
I20260929 14:11:16.803978 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet 4040985909414481ad0d1491249e8e67 with delete_type TABLET_DATA_TOMBSTONED (TS c5ec116e39144881bdf1f2cd3e923cf5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50768
I20260929 14:11:16.814513 6007 tablet_replica.cc:330] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5: stopping tablet replica
I20260929 14:11:16.823652 6007 raft_consensus.cc:2229] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:16.841720 6007 raft_consensus.cc:2258] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:16.847020 6007 ts_tablet_manager.cc:1905] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260929 14:11:16.870783 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 67356.1us
I20260929 14:11:16.900030 5344 raft_consensus.cc:1261] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:16.907929 6007 ts_tablet_manager.cc:1918] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:16.908586 6007 log.cc:1196] T 4040985909414481ad0d1491249e8e67 P c5ec116e39144881bdf1f2cd3e923cf5: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-1-root/wals/4040985909414481ad0d1491249e8e67
I20260929 14:11:16.955832 5084 catalog_manager.cc:4841] TS c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): tablet 4040985909414481ad0d1491249e8e67 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:16.967447 5868 consensus_queue.cc:1035] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:16.983978 5906 raft_consensus.cc:1261] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:17.002635 5197 raft_consensus.cc:1261] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:17.004520 5906 raft_consensus.cc:1261] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:17.017344 5868 consensus_queue.cc:1035] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:17.023296 5957 consensus_queue.cc:1035] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:17.046739 5868 raft_consensus.cc:2932] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:17.051580 6005 raft_consensus.cc:2932] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:17.053967 5905 raft_consensus.cc:2932] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:17.064148 5906 raft_consensus.cc:2932] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:17.076203 5099 catalog_manager.cc:5495] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:17.101202 5346 raft_consensus.cc:2932] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:17.107555 5096 catalog_manager.cc:5495] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:17.140427 5344 raft_consensus.cc:2932] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:17.142405 5269 consensus_queue.cc:237] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:17.148882 5906 raft_consensus.cc:1261] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:17.150434 5949 consensus_queue.cc:1035] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260929 14:11:17.157873 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 82992.6us
I20260929 14:11:17.160071 5350 raft_consensus.cc:1261] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:17.167059 5201 raft_consensus.cc:1261] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:17.168491 6005 raft_consensus.cc:2932] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:17.179206 5906 raft_consensus.cc:2932] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:17.187582 6006 consensus_queue.cc:1035] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:17.185654 5949 consensus_queue.cc:1035] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:11:17.198374 5350 raft_consensus.cc:2932] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:17.208022 5084 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 6978e56e8f49441795cc7f6b6a8f4cfe with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:17.236176 5096 catalog_manager.cc:5495] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260929 14:11:17.265548 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 77855.3us
I20260929 14:11:17.273063 5183 tablet_service.cc:1514] Processing DeleteTablet for tablet 6978e56e8f49441795cc7f6b6a8f4cfe with delete_type TABLET_DATA_TOMBSTONED (TS ac17d3b553b74c9da39e0ab307addacd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52180
I20260929 14:11:17.331804 6016 tablet_replica.cc:330] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd: stopping tablet replica
I20260929 14:11:17.341820 5197 raft_consensus.cc:2932] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:17.349643 6016 raft_consensus.cc:2229] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:17.350354 6016 raft_consensus.cc:2258] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:17.389976 6016 ts_tablet_manager.cc:1905] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:17.414438 6016 ts_tablet_manager.cc:1918] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.2
I20260929 14:11:17.414862 6016 log.cc:1196] T 6978e56e8f49441795cc7f6b6a8f4cfe P ac17d3b553b74c9da39e0ab307addacd: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-0-root/wals/6978e56e8f49441795cc7f6b6a8f4cfe
I20260929 14:11:17.416512 5086 catalog_manager.cc:4841] TS ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): tablet 6978e56e8f49441795cc7f6b6a8f4cfe (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:17.589138 5267 consensus_queue.cc:237] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:17.595927 5906 raft_consensus.cc:1261] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:17.597268 5868 consensus_queue.cc:1035] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:17.598925 5197 raft_consensus.cc:1261] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:17.600665 5868 consensus_queue.cc:1035] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:17.635078 5949 raft_consensus.cc:2932] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:17.637516 5906 raft_consensus.cc:2932] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:17.650806 5096 catalog_manager.cc:5495] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:17.656781 5200 raft_consensus.cc:2932] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:17.664863 5337 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e230699b3fb4d4d97fdb04988babce5 with delete_type TABLET_DATA_TOMBSTONED (TS d886fb1eb0474d2d9f7410c4013c45e4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47246
I20260929 14:11:17.669860 5084 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 5e230699b3fb4d4d97fdb04988babce5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:17.708788 6018 tablet_replica.cc:330] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4: stopping tablet replica
I20260929 14:11:17.710109 6018 raft_consensus.cc:2229] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:17.710932 6018 raft_consensus.cc:2258] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:17.716419 6018 ts_tablet_manager.cc:1905] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:17.733860 6018 ts_tablet_manager.cc:1918] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:17.734308 6018 log.cc:1196] T 5e230699b3fb4d4d97fdb04988babce5 P d886fb1eb0474d2d9f7410c4013c45e4: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-2-root/wals/5e230699b3fb4d4d97fdb04988babce5
I20260929 14:11:17.735812 5083 catalog_manager.cc:4841] TS d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): tablet 5e230699b3fb4d4d97fdb04988babce5 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:18.769439 5201 consensus_queue.cc:237] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:18.776306 5347 raft_consensus.cc:1261] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:18.777573 5947 consensus_queue.cc:1035] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260929 14:11:18.779947 5167 consensus_peers.cc:487] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e4a096fb48c46dd95a1d74d34b6e670. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:18.781980 5269 raft_consensus.cc:1261] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:18.784155 5951 consensus_queue.cc:1035] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:18.797946 5951 raft_consensus.cc:2932] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:18.809650 5344 raft_consensus.cc:2932] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:18.839123 5270 consensus_queue.cc:237] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:18.837908 5096 catalog_manager.cc:5495] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:18.841087 5269 raft_consensus.cc:2932] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:18.870615 5201 raft_consensus.cc:1261] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:18.870973 5344 raft_consensus.cc:1261] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 14:11:18.871682 5242 consensus_peers.cc:487] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88e8219d06734c04acdce83c7b9c1c53. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:18.873965 6019 consensus_queue.cc:1035] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:18.875707 6019 consensus_queue.cc:1035] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:18.890547 5347 raft_consensus.cc:2932] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:18.886364 5957 raft_consensus.cc:2932] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:18.902364 5201 raft_consensus.cc:2932] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:19.206122 5200 consensus_queue.cc:237] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:19.212926 5269 raft_consensus.cc:1261] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 14:11:19.213521 5167 consensus_peers.cc:487] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4300e83384e9458bb280a4b1e3131932. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:19.214244 5345 raft_consensus.cc:1261] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:19.215762 6035 consensus_queue.cc:1035] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:19.231617 6033 consensus_queue.cc:1035] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:19.242144 6041 ts_tablet_manager.cc:927] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147)
I20260929 14:11:19.244005 6041 tablet_copy_client.cc:316] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.65:45147
I20260929 14:11:19.263371 5097 catalog_manager.cc:5495] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:19.304570 5211 tablet_copy_service.cc:140] P ac17d3b553b74c9da39e0ab307addacd: Received BeginTabletCopySession request for tablet 0e4a096fb48c46dd95a1d74d34b6e670 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:37270)
I20260929 14:11:19.305147 5211 tablet_copy_service.cc:161] P ac17d3b553b74c9da39e0ab307addacd: Beginning new tablet copy session on tablet 0e4a096fb48c46dd95a1d74d34b6e670 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:37270: session id = 675b77a0a88a4bf08134281fc0cd949e-0e4a096fb48c46dd95a1d74d34b6e670
I20260929 14:11:19.314076 5211 tablet_copy_source_session.cc:205] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:19.317850 6041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e4a096fb48c46dd95a1d74d34b6e670. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:19.334403 6041 tablet_copy_client.cc:791] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:19.334992 6041 tablet_copy_client.cc:655] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:19.333572 6023 raft_consensus.cc:2932] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:19.338961 6041 tablet_copy_client.cc:531] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:19.342559 5345 raft_consensus.cc:2932] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:19.348433 6041 tablet_bootstrap.cc:492] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:19.408722 6041 tablet_bootstrap.cc:492] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:19.410933 6041 tablet_bootstrap.cc:492] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:19.412495 6041 ts_tablet_manager.cc:1397] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.064s user 0.021s sys 0.007s
I20260929 14:11:19.424947 6041 raft_consensus.cc:348] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:19.432771 6041 raft_consensus.cc:729] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:19.434309 6041 consensus_queue.cc:260] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:19.439236 5269 raft_consensus.cc:2932] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:19.476176 6041 ts_tablet_manager.cc:1428] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.063s user 0.006s sys 0.017s
I20260929 14:11:19.490279 5211 tablet_copy_service.cc:342] P ac17d3b553b74c9da39e0ab307addacd: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-0e4a096fb48c46dd95a1d74d34b6e670 received from {username='slave'} at 127.0.0.1:37270
I20260929 14:11:19.432648 5271 consensus_queue.cc:237] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:19.490867 5211 tablet_copy_service.cc:433] P ac17d3b553b74c9da39e0ab307addacd: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-0e4a096fb48c46dd95a1d74d34b6e670 on tablet 0e4a096fb48c46dd95a1d74d34b6e670 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:19.508301 6041 ts_tablet_manager.cc:927] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:19.512689 5097 catalog_manager.cc:5495] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:19.537648 5345 raft_consensus.cc:1261] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:19.539438 5868 consensus_queue.cc:1035] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260929 14:11:19.566219 5242 consensus_peers.cc:487] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e55a9c8ae48e44c59fee506e539b3f90. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:19.573792 5200 raft_consensus.cc:1261] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:19.588325 5868 raft_consensus.cc:2932] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:19.590926 5345 raft_consensus.cc:2932] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:19.601843 6031 consensus_queue.cc:1035] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:19.644335 6041 tablet_copy_client.cc:316] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.66:43473
I20260929 14:11:19.646479 5286 tablet_copy_service.cc:140] P c5ec116e39144881bdf1f2cd3e923cf5: Received BeginTabletCopySession request for tablet 88e8219d06734c04acdce83c7b9c1c53 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:41626)
I20260929 14:11:19.647100 5286 tablet_copy_service.cc:161] P c5ec116e39144881bdf1f2cd3e923cf5: Beginning new tablet copy session on tablet 88e8219d06734c04acdce83c7b9c1c53 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:41626: session id = 675b77a0a88a4bf08134281fc0cd949e-88e8219d06734c04acdce83c7b9c1c53
I20260929 14:11:19.661917 5097 catalog_manager.cc:5495] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 2 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:19.668226 6054 ts_tablet_manager.cc:927] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147)
I20260929 14:11:19.676347 5200 raft_consensus.cc:2932] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:19.764197 5286 tablet_copy_source_session.cc:205] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:19.767829 6041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88e8219d06734c04acdce83c7b9c1c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:19.770501 6054 tablet_copy_client.cc:316] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.65:45147
I20260929 14:11:19.779882 5211 tablet_copy_service.cc:140] P ac17d3b553b74c9da39e0ab307addacd: Received BeginTabletCopySession request for tablet 4300e83384e9458bb280a4b1e3131932 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:37270)
I20260929 14:11:19.780423 5211 tablet_copy_service.cc:161] P ac17d3b553b74c9da39e0ab307addacd: Beginning new tablet copy session on tablet 4300e83384e9458bb280a4b1e3131932 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:37270: session id = 675b77a0a88a4bf08134281fc0cd949e-4300e83384e9458bb280a4b1e3131932
I20260929 14:11:19.812211 5211 tablet_copy_source_session.cc:205] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:19.815447 6054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4300e83384e9458bb280a4b1e3131932. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:19.819453 6041 tablet_copy_client.cc:791] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:19.824497 6041 tablet_copy_client.cc:655] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:19.832007 6041 tablet_copy_client.cc:531] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:19.833503 5906 raft_consensus.cc:1203] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:19.840152 6054 tablet_copy_client.cc:791] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:19.845681 6054 tablet_copy_client.cc:655] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:19.844961 6041 tablet_bootstrap.cc:492] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:19.854789 6054 tablet_copy_client.cc:531] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:19.863751 6054 tablet_bootstrap.cc:492] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:19.872464 6041 tablet_bootstrap.cc:492] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:19.873397 6041 tablet_bootstrap.cc:492] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:19.874053 6041 ts_tablet_manager.cc:1397] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.014s
I20260929 14:11:19.876978 6041 raft_consensus.cc:348] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:19.878329 6041 raft_consensus.cc:729] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:19.879238 6041 consensus_queue.cc:260] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:19.882079 6041 ts_tablet_manager.cc:1428] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.008s user 0.002s sys 0.005s
I20260929 14:11:19.883585 5286 tablet_copy_service.cc:342] P c5ec116e39144881bdf1f2cd3e923cf5: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-88e8219d06734c04acdce83c7b9c1c53 received from {username='slave'} at 127.0.0.1:41626
I20260929 14:11:19.883949 5286 tablet_copy_service.cc:433] P c5ec116e39144881bdf1f2cd3e923cf5: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-88e8219d06734c04acdce83c7b9c1c53 on tablet 88e8219d06734c04acdce83c7b9c1c53 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:19.892525 6054 tablet_bootstrap.cc:492] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:19.893685 6054 tablet_bootstrap.cc:492] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:19.894325 6054 ts_tablet_manager.cc:1397] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.004s
I20260929 14:11:19.896607 6054 raft_consensus.cc:348] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:19.897176 6054 raft_consensus.cc:729] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:19.897817 6054 consensus_queue.cc:260] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:19.899534 6054 ts_tablet_manager.cc:1428] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 14:11:19.901093 5211 tablet_copy_service.cc:342] P ac17d3b553b74c9da39e0ab307addacd: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-4300e83384e9458bb280a4b1e3131932 received from {username='slave'} at 127.0.0.1:37270
I20260929 14:11:19.901500 5211 tablet_copy_service.cc:433] P ac17d3b553b74c9da39e0ab307addacd: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-4300e83384e9458bb280a4b1e3131932 on tablet 4300e83384e9458bb280a4b1e3131932 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:19.995641 6054 ts_tablet_manager.cc:927] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:19.997697 6054 tablet_copy_client.cc:316] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.66:43473
I20260929 14:11:19.999513 5286 tablet_copy_service.cc:140] P c5ec116e39144881bdf1f2cd3e923cf5: Received BeginTabletCopySession request for tablet e55a9c8ae48e44c59fee506e539b3f90 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:41626)
I20260929 14:11:20.000087 5286 tablet_copy_service.cc:161] P c5ec116e39144881bdf1f2cd3e923cf5: Beginning new tablet copy session on tablet e55a9c8ae48e44c59fee506e539b3f90 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:41626: session id = 675b77a0a88a4bf08134281fc0cd949e-e55a9c8ae48e44c59fee506e539b3f90
I20260929 14:11:20.007046 5286 tablet_copy_source_session.cc:205] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:20.009994 6054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e55a9c8ae48e44c59fee506e539b3f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:20.022091 6054 tablet_copy_client.cc:791] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:20.022696 6054 tablet_copy_client.cc:655] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:20.026427 6054 tablet_copy_client.cc:531] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:20.034116 6054 tablet_bootstrap.cc:492] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:20.055014 6054 tablet_bootstrap.cc:492] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:20.056167 6054 tablet_bootstrap.cc:492] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:20.056937 6054 ts_tablet_manager.cc:1397] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.003s
I20260929 14:11:20.059793 6054 raft_consensus.cc:348] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:20.060815 6054 raft_consensus.cc:729] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:20.078106 6054 consensus_queue.cc:260] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
W20260929 14:11:20.078903 5297 debug-util.cc:398] Leaking SignalData structure 0x7b0800096d60 after lost signal to thread 4044
W20260929 14:11:20.080559 5297 debug-util.cc:398] Leaking SignalData structure 0x7b0800056300 after lost signal to thread 5133
W20260929 14:11:20.082223 5297 debug-util.cc:398] Leaking SignalData structure 0x7b0800168580 after lost signal to thread 5225
W20260929 14:11:20.083603 5297 debug-util.cc:398] Leaking SignalData structure 0x7b0800237860 after lost signal to thread 5300
W20260929 14:11:20.085050 5297 debug-util.cc:398] Leaking SignalData structure 0x7b08000ce640 after lost signal to thread 5375
W20260929 14:11:20.086683 5297 debug-util.cc:398] Leaking SignalData structure 0x7b08002e6e00 after lost signal to thread 5932
I20260929 14:11:20.091231 6054 ts_tablet_manager.cc:1428] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.034s user 0.005s sys 0.000s
I20260929 14:11:20.093389 5286 tablet_copy_service.cc:342] P c5ec116e39144881bdf1f2cd3e923cf5: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-e55a9c8ae48e44c59fee506e539b3f90 received from {username='slave'} at 127.0.0.1:41626
I20260929 14:11:20.093827 5286 tablet_copy_service.cc:433] P c5ec116e39144881bdf1f2cd3e923cf5: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-e55a9c8ae48e44c59fee506e539b3f90 on tablet e55a9c8ae48e44c59fee506e539b3f90 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:20.121779 5906 raft_consensus.cc:1203] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:20.200414 5906 raft_consensus.cc:1203] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:20.285709 5951 raft_consensus.cc:1050] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:20.290853 5951 consensus_queue.cc:237] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:20.298744 5906 raft_consensus.cc:1261] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:20.300266 6033 consensus_queue.cc:1035] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:20.307714 5272 raft_consensus.cc:1261] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:20.310250 6033 consensus_queue.cc:1035] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:20.314906 5345 raft_consensus.cc:1261] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:20.316401 6033 consensus_queue.cc:1035] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:20.334908 5951 raft_consensus.cc:2932] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.345111 5905 raft_consensus.cc:2932] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.370975 5347 raft_consensus.cc:2932] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.389103 5906 raft_consensus.cc:1203] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260929 14:11:20.420059 5097 catalog_manager.cc:5495] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.450466 5267 raft_consensus.cc:2932] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
W20260929 14:11:20.456801 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 84912us
I20260929 14:11:20.487537 5197 consensus_queue.cc:237] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:20.496718 5345 raft_consensus.cc:1261] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:20.498289 5905 raft_consensus.cc:1261] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:20.499981 6066 consensus_queue.cc:1035] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:20.502640 6033 consensus_queue.cc:1035] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:20.506174 5951 raft_consensus.cc:2932] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.514125 5086 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 0e4a096fb48c46dd95a1d74d34b6e670 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:20.514506 5905 raft_consensus.cc:2932] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.520802 5344 raft_consensus.cc:2932] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.535692 5097 catalog_manager.cc:5495] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:20.553174 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e4a096fb48c46dd95a1d74d34b6e670 with delete_type TABLET_DATA_TOMBSTONED (TS c5ec116e39144881bdf1f2cd3e923cf5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50768
I20260929 14:11:20.557272 6071 tablet_replica.cc:330] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5: stopping tablet replica
I20260929 14:11:20.558379 6071 raft_consensus.cc:2229] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:20.559007 6071 raft_consensus.cc:2258] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:20.561851 6071 ts_tablet_manager.cc:1905] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:20.575209 6071 ts_tablet_manager.cc:1918] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:20.575610 6071 log.cc:1196] T 0e4a096fb48c46dd95a1d74d34b6e670 P c5ec116e39144881bdf1f2cd3e923cf5: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-1-root/wals/0e4a096fb48c46dd95a1d74d34b6e670
I20260929 14:11:20.577049 5084 catalog_manager.cc:4841] TS c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): tablet 0e4a096fb48c46dd95a1d74d34b6e670 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:20.736171 5868 raft_consensus.cc:1050] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:20.738435 5868 consensus_queue.cc:237] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:20.742151 6050 raft_consensus.cc:1050] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:20.744470 6050 consensus_queue.cc:237] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:20.751153 5905 raft_consensus.cc:1261] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:20.751920 5344 raft_consensus.cc:1261] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:20.753515 6066 consensus_queue.cc:1035] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:20.754011 5905 raft_consensus.cc:1261] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:20.755324 6066 consensus_queue.cc:1035] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:20.757483 5344 raft_consensus.cc:1261] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:20.757617 5868 consensus_queue.cc:1035] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:20.758754 5200 raft_consensus.cc:1261] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:20.758878 6031 consensus_queue.cc:1035] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:20.760623 6031 consensus_queue.cc:1035] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:20.762810 5267 raft_consensus.cc:1261] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:20.764365 6050 consensus_queue.cc:1035] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:20.783471 6065 raft_consensus.cc:2932] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.793418 5346 raft_consensus.cc:2932] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.799005 5267 raft_consensus.cc:2932] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.791893 6019 raft_consensus.cc:2932] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.790472 5347 raft_consensus.cc:2932] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.811614 5200 raft_consensus.cc:2932] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.810178 5905 raft_consensus.cc:2932] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.831929 5097 catalog_manager.cc:5495] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:20.853034 5097 catalog_manager.cc:5495] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.854785 5200 consensus_queue.cc:237] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:20.858911 5906 raft_consensus.cc:2932] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.866516 5267 raft_consensus.cc:1261] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:20.870574 5907 raft_consensus.cc:1261] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:20.876564 6065 consensus_queue.cc:1035] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:11:20.877629 6050 consensus_queue.cc:1035] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:20.877974 5271 consensus_queue.cc:237] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:20.889463 5347 raft_consensus.cc:1261] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:20.891695 6031 consensus_queue.cc:1035] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:20.900621 5906 raft_consensus.cc:1261] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:20.900430 5868 raft_consensus.cc:2932] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.902313 5951 raft_consensus.cc:2932] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.903728 5906 raft_consensus.cc:2932] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.908952 6031 consensus_queue.cc:1035] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:20.909986 5086 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4300e83384e9458bb280a4b1e3131932 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:20.913012 5907 raft_consensus.cc:2932] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.914635 5084 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 88e8219d06734c04acdce83c7b9c1c53 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:20.915503 5347 raft_consensus.cc:2932] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.919200 5272 raft_consensus.cc:2932] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.921916 5098 catalog_manager.cc:5495] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:20.933677 5099 catalog_manager.cc:5495] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:20.949626 5337 tablet_service.cc:1514] Processing DeleteTablet for tablet 4300e83384e9458bb280a4b1e3131932 with delete_type TABLET_DATA_TOMBSTONED (TS d886fb1eb0474d2d9f7410c4013c45e4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47246
I20260929 14:11:20.953094 5183 tablet_service.cc:1514] Processing DeleteTablet for tablet 88e8219d06734c04acdce83c7b9c1c53 with delete_type TABLET_DATA_TOMBSTONED (TS ac17d3b553b74c9da39e0ab307addacd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52180
I20260929 14:11:20.981597 6081 tablet_replica.cc:330] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4: stopping tablet replica
I20260929 14:11:20.983632 6082 tablet_replica.cc:330] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd: stopping tablet replica
I20260929 14:11:20.985615 6082 raft_consensus.cc:2229] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:20.986310 6082 raft_consensus.cc:2258] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:20.989897 6082 ts_tablet_manager.cc:1905] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:20.993403 6031 raft_consensus.cc:1050] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:20.994997 6081 raft_consensus.cc:2229] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:20.999400 6081 raft_consensus.cc:2258] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:21.007609 6031 consensus_queue.cc:237] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:21.014706 6082 ts_tablet_manager.cc:1918] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:21.015242 6082 log.cc:1196] T 88e8219d06734c04acdce83c7b9c1c53 P ac17d3b553b74c9da39e0ab307addacd: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-0-root/wals/88e8219d06734c04acdce83c7b9c1c53
I20260929 14:11:21.016786 5086 catalog_manager.cc:4841] TS ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): tablet 88e8219d06734c04acdce83c7b9c1c53 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:21.019552 6081 ts_tablet_manager.cc:1905] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:21.024912 5906 raft_consensus.cc:1261] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEARNER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260929 14:11:21.025738 5347 raft_consensus.cc:1261] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260929 14:11:21.026603 6019 consensus_queue.cc:1035] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:21.028569 5957 consensus_queue.cc:1035] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:21.035434 6081 ts_tablet_manager.cc:1918] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:21.035940 6081 log.cc:1196] T 4300e83384e9458bb280a4b1e3131932 P d886fb1eb0474d2d9f7410c4013c45e4: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-2-root/wals/4300e83384e9458bb280a4b1e3131932
I20260929 14:11:21.039207 5200 raft_consensus.cc:1261] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260929 14:11:21.041813 5957 consensus_queue.cc:1035] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:21.043246 5083 catalog_manager.cc:4841] TS d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): tablet 4300e83384e9458bb280a4b1e3131932 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:21.058024 5957 raft_consensus.cc:2932] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:21.059739 5906 raft_consensus.cc:2932] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:21.061774 5197 raft_consensus.cc:2932] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:21.069731 5347 raft_consensus.cc:2932] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:21.074931 5097 catalog_manager.cc:5495] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:21.102909 5272 consensus_queue.cc:237] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:21.107609 5347 raft_consensus.cc:1261] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260929 14:11:21.108945 5957 consensus_queue.cc:1035] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:21.109630 5906 raft_consensus.cc:1261] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260929 14:11:21.111380 6019 consensus_queue.cc:1035] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:21.117501 5868 raft_consensus.cc:2932] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:21.119555 5347 raft_consensus.cc:2932] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:21.121961 5907 raft_consensus.cc:2932] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:21.135327 5084 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e55a9c8ae48e44c59fee506e539b3f90 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:21.136337 5096 catalog_manager.cc:5495] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New cstate: current_term: 2 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:21.148572 5183 tablet_service.cc:1514] Processing DeleteTablet for tablet e55a9c8ae48e44c59fee506e539b3f90 with delete_type TABLET_DATA_TOMBSTONED (TS ac17d3b553b74c9da39e0ab307addacd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52180
I20260929 14:11:21.149470 6082 tablet_replica.cc:330] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd: stopping tablet replica
I20260929 14:11:21.150275 6082 raft_consensus.cc:2229] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:21.150844 6082 raft_consensus.cc:2258] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:21.153710 6082 ts_tablet_manager.cc:1905] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:21.166970 6082 ts_tablet_manager.cc:1918] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260929 14:11:21.167295 6082 log.cc:1196] T e55a9c8ae48e44c59fee506e539b3f90 P ac17d3b553b74c9da39e0ab307addacd: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-0-root/wals/e55a9c8ae48e44c59fee506e539b3f90
I20260929 14:11:21.168741 5086 catalog_manager.cc:4841] TS ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): tablet e55a9c8ae48e44c59fee506e539b3f90 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:22.215256 5345 consensus_queue.cc:237] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
W20260929 14:11:22.221814 5316 consensus_peers.cc:487] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae31abec82ff4f40b695ccb19f289dbc. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:22.222904 5271 raft_consensus.cc:1261] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:22.224291 5200 raft_consensus.cc:1261] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:22.224328 6088 consensus_queue.cc:1035] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:22.225666 6038 consensus_queue.cc:1035] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:22.233678 6088 raft_consensus.cc:2932] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:22.235437 5197 raft_consensus.cc:2932] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:22.235605 5272 raft_consensus.cc:2932] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:22.248014 5200 consensus_queue.cc:237] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:22.251230 5096 catalog_manager.cc:5495] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260929 14:11:22.256544 5167 consensus_peers.cc:487] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a38ce5b8e5024c0ab37787aaf2b357c0. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:22.257530 5271 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:22.257877 5345 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:22.259033 5951 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:22.260614 5951 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:22.286440 6087 raft_consensus.cc:2932] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:22.287894 5267 raft_consensus.cc:2932] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:22.312780 5346 consensus_queue.cc:237] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:22.316658 5345 raft_consensus.cc:2932] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:22.327812 5267 raft_consensus.cc:1261] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:22.329280 6039 consensus_queue.cc:1035] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:22.329543 5099 catalog_manager.cc:5495] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:22.330672 5200 raft_consensus.cc:1261] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 14:11:22.333679 5316 consensus_peers.cc:487] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94e36a01dad24c1193b628e9200d3c2d. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:22.341672 6039 consensus_queue.cc:1035] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260929 14:11:22.372357 6088 raft_consensus.cc:2932] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:22.376466 5272 raft_consensus.cc:2932] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:22.382169 5199 raft_consensus.cc:2932] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:22.393049 5271 consensus_queue.cc:237] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:22.401947 5099 catalog_manager.cc:5495] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:22.425071 5345 raft_consensus.cc:1261] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 14:11:22.425500 5242 consensus_peers.cc:487] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 64181b973d25469da29a63928ede710f. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:22.425071 5200 raft_consensus.cc:1261] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:22.427395 5868 consensus_queue.cc:1035] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:22.429355 6104 consensus_queue.cc:1035] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:22.437067 5868 raft_consensus.cc:2932] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:22.439149 5200 raft_consensus.cc:2932] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:22.445638 5345 raft_consensus.cc:2932] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:22.457381 5099 catalog_manager.cc:5495] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:22.673695 6108 ts_tablet_manager.cc:927] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:22.675669 6108 tablet_copy_client.cc:316] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.67:42639
I20260929 14:11:22.677791 5361 tablet_copy_service.cc:140] P d886fb1eb0474d2d9f7410c4013c45e4: Received BeginTabletCopySession request for tablet ae31abec82ff4f40b695ccb19f289dbc from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:38536)
I20260929 14:11:22.678293 5361 tablet_copy_service.cc:161] P d886fb1eb0474d2d9f7410c4013c45e4: Beginning new tablet copy session on tablet ae31abec82ff4f40b695ccb19f289dbc from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:38536: session id = 675b77a0a88a4bf08134281fc0cd949e-ae31abec82ff4f40b695ccb19f289dbc
I20260929 14:11:22.685882 5361 tablet_copy_source_session.cc:205] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:22.689314 6108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae31abec82ff4f40b695ccb19f289dbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:22.703570 6108 tablet_copy_client.cc:791] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:22.704133 6108 tablet_copy_client.cc:655] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:22.707370 6108 tablet_copy_client.cc:531] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:22.715052 6108 tablet_bootstrap.cc:492] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:22.734189 6108 tablet_bootstrap.cc:492] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:22.735051 6108 tablet_bootstrap.cc:492] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:22.735672 6108 ts_tablet_manager.cc:1397] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.005s
I20260929 14:11:22.738044 6108 raft_consensus.cc:348] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:22.738641 6108 raft_consensus.cc:729] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:22.739239 6108 consensus_queue.cc:260] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:22.742437 6108 ts_tablet_manager.cc:1428] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260929 14:11:22.745879 5361 tablet_copy_service.cc:342] P d886fb1eb0474d2d9f7410c4013c45e4: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-ae31abec82ff4f40b695ccb19f289dbc received from {username='slave'} at 127.0.0.1:38536
I20260929 14:11:22.746250 5361 tablet_copy_service.cc:433] P d886fb1eb0474d2d9f7410c4013c45e4: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-ae31abec82ff4f40b695ccb19f289dbc on tablet ae31abec82ff4f40b695ccb19f289dbc with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:22.751504 6108 ts_tablet_manager.cc:927] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:22.765652 6111 ts_tablet_manager.cc:927] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147)
I20260929 14:11:22.769542 6108 tablet_copy_client.cc:316] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.67:42639
I20260929 14:11:22.772029 5361 tablet_copy_service.cc:140] P d886fb1eb0474d2d9f7410c4013c45e4: Received BeginTabletCopySession request for tablet 94e36a01dad24c1193b628e9200d3c2d from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:38536)
I20260929 14:11:22.772552 5361 tablet_copy_service.cc:161] P d886fb1eb0474d2d9f7410c4013c45e4: Beginning new tablet copy session on tablet 94e36a01dad24c1193b628e9200d3c2d from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:38536: session id = 675b77a0a88a4bf08134281fc0cd949e-94e36a01dad24c1193b628e9200d3c2d
I20260929 14:11:22.775344 6111 tablet_copy_client.cc:316] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.65:45147
I20260929 14:11:22.776999 5211 tablet_copy_service.cc:140] P ac17d3b553b74c9da39e0ab307addacd: Received BeginTabletCopySession request for tablet a38ce5b8e5024c0ab37787aaf2b357c0 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:37270)
I20260929 14:11:22.777549 5211 tablet_copy_service.cc:161] P ac17d3b553b74c9da39e0ab307addacd: Beginning new tablet copy session on tablet a38ce5b8e5024c0ab37787aaf2b357c0 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:37270: session id = 675b77a0a88a4bf08134281fc0cd949e-a38ce5b8e5024c0ab37787aaf2b357c0
I20260929 14:11:22.782052 5361 tablet_copy_source_session.cc:205] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:22.785678 6108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e36a01dad24c1193b628e9200d3c2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:22.787995 5211 tablet_copy_source_session.cc:205] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:22.791976 6111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a38ce5b8e5024c0ab37787aaf2b357c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:22.801316 6108 tablet_copy_client.cc:791] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:22.802125 6108 tablet_copy_client.cc:655] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:22.805981 6108 tablet_copy_client.cc:531] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:22.807444 6111 tablet_copy_client.cc:791] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:22.807992 6111 tablet_copy_client.cc:655] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:22.811442 6111 tablet_copy_client.cc:531] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:22.815613 6108 tablet_bootstrap.cc:492] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:22.821674 6111 tablet_bootstrap.cc:492] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:22.841580 6117 ts_tablet_manager.cc:927] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:22.845975 6111 tablet_bootstrap.cc:492] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:22.847112 6111 tablet_bootstrap.cc:492] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:22.848083 6111 ts_tablet_manager.cc:1397] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.027s user 0.014s sys 0.010s
I20260929 14:11:22.851500 6111 raft_consensus.cc:348] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:22.852645 6111 raft_consensus.cc:729] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:22.854864 6111 consensus_queue.cc:260] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:22.858073 6111 ts_tablet_manager.cc:1428] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.010s user 0.006s sys 0.001s
I20260929 14:11:22.863426 5211 tablet_copy_service.cc:342] P ac17d3b553b74c9da39e0ab307addacd: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-a38ce5b8e5024c0ab37787aaf2b357c0 received from {username='slave'} at 127.0.0.1:37270
I20260929 14:11:22.863873 5211 tablet_copy_service.cc:433] P ac17d3b553b74c9da39e0ab307addacd: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-a38ce5b8e5024c0ab37787aaf2b357c0 on tablet a38ce5b8e5024c0ab37787aaf2b357c0 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:22.873502 6117 tablet_copy_client.cc:316] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.66:43473
I20260929 14:11:22.876187 5286 tablet_copy_service.cc:140] P c5ec116e39144881bdf1f2cd3e923cf5: Received BeginTabletCopySession request for tablet 64181b973d25469da29a63928ede710f from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:41626)
I20260929 14:11:22.876794 5286 tablet_copy_service.cc:161] P c5ec116e39144881bdf1f2cd3e923cf5: Beginning new tablet copy session on tablet 64181b973d25469da29a63928ede710f from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:41626: session id = 675b77a0a88a4bf08134281fc0cd949e-64181b973d25469da29a63928ede710f
I20260929 14:11:22.879981 6108 tablet_bootstrap.cc:492] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:22.880918 6108 tablet_bootstrap.cc:492] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:22.881827 6108 ts_tablet_manager.cc:1397] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.066s user 0.023s sys 0.019s
I20260929 14:11:22.885196 5286 tablet_copy_source_session.cc:205] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:22.884689 6108 raft_consensus.cc:348] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:22.886538 6108 raft_consensus.cc:729] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:22.887311 6108 consensus_queue.cc:260] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:22.888294 6117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64181b973d25469da29a63928ede710f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:22.889832 6108 ts_tablet_manager.cc:1428] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260929 14:11:22.891674 5361 tablet_copy_service.cc:342] P d886fb1eb0474d2d9f7410c4013c45e4: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-94e36a01dad24c1193b628e9200d3c2d received from {username='slave'} at 127.0.0.1:38536
I20260929 14:11:22.892048 5361 tablet_copy_service.cc:433] P d886fb1eb0474d2d9f7410c4013c45e4: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-94e36a01dad24c1193b628e9200d3c2d on tablet 94e36a01dad24c1193b628e9200d3c2d with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:22.904599 6117 tablet_copy_client.cc:791] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:22.905148 6117 tablet_copy_client.cc:655] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:22.908547 6117 tablet_copy_client.cc:531] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:22.916603 6117 tablet_bootstrap.cc:492] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:22.936493 6117 tablet_bootstrap.cc:492] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:22.937278 6117 tablet_bootstrap.cc:492] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:22.937990 6117 ts_tablet_manager.cc:1397] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.012s
I20260929 14:11:22.940616 6117 raft_consensus.cc:348] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:22.941370 6117 raft_consensus.cc:729] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:22.941989 6117 consensus_queue.cc:260] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:22.947204 6117 ts_tablet_manager.cc:1428] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.009s user 0.001s sys 0.004s
I20260929 14:11:22.953142 5286 tablet_copy_service.cc:342] P c5ec116e39144881bdf1f2cd3e923cf5: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-64181b973d25469da29a63928ede710f received from {username='slave'} at 127.0.0.1:41626
I20260929 14:11:22.953603 5286 tablet_copy_service.cc:433] P c5ec116e39144881bdf1f2cd3e923cf5: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-64181b973d25469da29a63928ede710f on tablet 64181b973d25469da29a63928ede710f with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:23.178186 5906 raft_consensus.cc:1203] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:23.237203 5906 raft_consensus.cc:1203] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:23.270511 5906 raft_consensus.cc:1203] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:23.308704 5906 raft_consensus.cc:1203] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:23.668478 6088 raft_consensus.cc:1050] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:23.670926 6088 consensus_queue.cc:237] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:23.677070 5907 raft_consensus.cc:1261] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:23.678269 6039 consensus_queue.cc:1035] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:23.678582 5271 raft_consensus.cc:1261] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:23.679764 6097 consensus_queue.cc:1035] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:23.690785 5200 raft_consensus.cc:1261] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:23.698833 6088 consensus_queue.cc:1035] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:23.698107 6039 raft_consensus.cc:2932] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:23.706310 5272 raft_consensus.cc:2932] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:23.714148 5907 raft_consensus.cc:2932] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:23.715152 5200 raft_consensus.cc:2932] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:23.728389 5096 catalog_manager.cc:5495] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:23.802547 5346 consensus_queue.cc:237] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:23.822271 5906 raft_consensus.cc:1261] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:23.823609 6121 raft_consensus.cc:1050] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:23.827273 6088 consensus_queue.cc:1035] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:11:23.829505 6121 consensus_queue.cc:237] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:23.845539 5906 raft_consensus.cc:2932] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:23.847659 6097 raft_consensus.cc:1050] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:23.850171 6097 consensus_queue.cc:237] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:23.843888 6088 raft_consensus.cc:2932] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:23.857424 5272 raft_consensus.cc:1261] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:23.861949 6132 consensus_queue.cc:1035] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:11:24.343150 5096 catalog_manager.cc:5495] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.372694 5272 raft_consensus.cc:2932] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.402424 6139 raft_consensus.cc:1050] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:24.403544 5083 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 94e36a01dad24c1193b628e9200d3c2d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:24.404621 5346 raft_consensus.cc:1261] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:24.431118 6139 consensus_queue.cc:237] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:24.428021 6121 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.006s sys 0.001s
I20260929 14:11:24.443428 5200 raft_consensus.cc:1261] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260929 14:11:24.443557 6121 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.550s user 0.007s sys 0.001s
I20260929 14:11:24.454708 5907 raft_consensus.cc:1261] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:24.466770 5905 raft_consensus.cc:1261] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:24.467334 5195 raft_consensus.cc:1261] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:24.478389 5868 consensus_queue.cc:1035] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:24.483422 6141 consensus_queue.cc:1035] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:24.519759 5269 raft_consensus.cc:1261] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260929 14:11:24.524487 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 73617.5us
I20260929 14:11:24.526672 6142 raft_consensus.cc:2932] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.530587 5195 raft_consensus.cc:2932] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.542508 6121 consensus_queue.cc:1035] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:24.546904 6132 consensus_queue.cc:1035] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:24.548170 5906 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:24.551647 5267 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:24.556454 6139 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260929 14:11:24.576157 6014 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:24.556238 6148 consensus_queue.cc:1035] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260929 14:11:24.615270 6088 consensus_queue.cc:1035] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260929 14:11:24.648301 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 47253.3us
I20260929 14:11:24.649641 5905 raft_consensus.cc:2932] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.677330 5350 raft_consensus.cc:2932] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.693964 5183 tablet_service.cc:1514] Processing DeleteTablet for tablet 94e36a01dad24c1193b628e9200d3c2d with delete_type TABLET_DATA_TOMBSTONED (TS ac17d3b553b74c9da39e0ab307addacd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52180
I20260929 14:11:24.697034 5099 catalog_manager.cc:5042] ChangeConfig:REMOVE_PEER RPC for tablet 94e36a01dad24c1193b628e9200d3c2d with cas_config_opid_index 3: aborting the task: latest config opid_index 4; task opid_index 3
I20260929 14:11:24.699143 5272 raft_consensus.cc:2932] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.698740 5097 catalog_manager.cc:5495] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:24.693614 6148 raft_consensus.cc:2932] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.720167 5351 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:24.726788 5272 consensus_queue.cc:237] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:24.760629 5200 raft_consensus.cc:2932] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.782397 6014 raft_consensus.cc:2932] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.798938 5267 raft_consensus.cc:2932] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.800307 5906 raft_consensus.cc:1261] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:24.797861 5097 catalog_manager.cc:5495] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.806813 5201 raft_consensus.cc:1261] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:24.812124 5868 consensus_queue.cc:1035] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260929 14:11:24.813773 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 89115.9us
I20260929 14:11:24.818982 6163 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:24.827479 5868 consensus_queue.cc:1035] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:24.827629 5906 raft_consensus.cc:2932] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.829131 6162 tablet_replica.cc:330] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd: stopping tablet replica
I20260929 14:11:24.830559 6162 raft_consensus.cc:2229] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:24.831419 6162 raft_consensus.cc:2258] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:24.843333 6131 raft_consensus.cc:2932] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.847247 6162 ts_tablet_manager.cc:1905] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:24.851923 5903 raft_consensus.cc:2932] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.862449 5201 raft_consensus.cc:2932] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.888629 5098 catalog_manager.cc:5495] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.891182 5907 raft_consensus.cc:2932] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.896082 5342 consensus_queue.cc:237] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:24.902724 5222 debug-util.cc:398] Leaking SignalData structure 0x7b0800122400 after lost signal to thread 4044
W20260929 14:11:24.903667 5222 debug-util.cc:398] Leaking SignalData structure 0x7b080009ca00 after lost signal to thread 5133
W20260929 14:11:24.904907 5222 debug-util.cc:398] Leaking SignalData structure 0x7b08000e3360 after lost signal to thread 5225
W20260929 14:11:24.905879 5222 debug-util.cc:398] Leaking SignalData structure 0x7b080012fbe0 after lost signal to thread 5300
W20260929 14:11:24.906836 5222 debug-util.cc:398] Leaking SignalData structure 0x7b08003064a0 after lost signal to thread 5375
I20260929 14:11:24.907302 5906 raft_consensus.cc:1261] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260929 14:11:24.907831 5222 debug-util.cc:398] Leaking SignalData structure 0x7b08002ecc80 after lost signal to thread 5932
I20260929 14:11:24.909036 6039 consensus_queue.cc:1035] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:24.911438 5200 raft_consensus.cc:1261] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:24.915937 6146 consensus_queue.cc:1035] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:11:24.920437 5344 raft_consensus.cc:2932] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:24.954582 5098 catalog_manager.cc:5495] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
W20260929 14:11:24.973023 6164 log.cc:924] Time spent T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20260929 14:11:24.988389 6162 ts_tablet_manager.cc:1918] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:24.989802 5337 tablet_service.cc:1514] Processing DeleteTablet for tablet 64181b973d25469da29a63928ede710f with delete_type TABLET_DATA_TOMBSTONED (TS d886fb1eb0474d2d9f7410c4013c45e4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47246
I20260929 14:11:24.992563 5084 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 64181b973d25469da29a63928ede710f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:25.004910 6162 log.cc:1196] T 94e36a01dad24c1193b628e9200d3c2d P ac17d3b553b74c9da39e0ab307addacd: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-0-root/wals/94e36a01dad24c1193b628e9200d3c2d
I20260929 14:11:25.007849 5086 catalog_manager.cc:4841] TS ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): tablet 94e36a01dad24c1193b628e9200d3c2d (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:25.031140 6146 raft_consensus.cc:2932] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:25.052232 5903 raft_consensus.cc:2932] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:25.066159 5083 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ae31abec82ff4f40b695ccb19f289dbc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:25.067195 6165 tablet_replica.cc:330] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4: stopping tablet replica
I20260929 14:11:25.073004 6165 raft_consensus.cc:2229] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:25.077821 5097 catalog_manager.cc:5495] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:25.080579 5199 raft_consensus.cc:2932] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:25.083252 6165 raft_consensus.cc:2258] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:25.086891 6165 ts_tablet_manager.cc:1905] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:25.115687 6165 ts_tablet_manager.cc:1918] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:25.117441 6165 log.cc:1196] T 64181b973d25469da29a63928ede710f P d886fb1eb0474d2d9f7410c4013c45e4: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-2-root/wals/64181b973d25469da29a63928ede710f
I20260929 14:11:25.120029 5083 catalog_manager.cc:4841] TS d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): tablet 64181b973d25469da29a63928ede710f (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:25.126397 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet ae31abec82ff4f40b695ccb19f289dbc with delete_type TABLET_DATA_TOMBSTONED (TS c5ec116e39144881bdf1f2cd3e923cf5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50768
I20260929 14:11:25.150202 6166 tablet_replica.cc:330] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5: stopping tablet replica
I20260929 14:11:25.152565 6166 raft_consensus.cc:2229] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:25.153568 6166 raft_consensus.cc:2258] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:25.157547 6166 ts_tablet_manager.cc:1905] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:25.176477 6166 ts_tablet_manager.cc:1918] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:25.176976 6166 log.cc:1196] T ae31abec82ff4f40b695ccb19f289dbc P c5ec116e39144881bdf1f2cd3e923cf5: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-1-root/wals/ae31abec82ff4f40b695ccb19f289dbc
I20260929 14:11:25.178512 5084 catalog_manager.cc:4841] TS c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): tablet ae31abec82ff4f40b695ccb19f289dbc (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:25.332466 5200 consensus_queue.cc:237] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:25.355459 5342 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:25.356877 6138 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:25.365875 5903 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:25.367555 6143 raft_consensus.cc:2932] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:25.380167 5350 raft_consensus.cc:2932] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:25.386647 6163 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:11:25.392494 5903 raft_consensus.cc:2932] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
W20260929 14:11:25.395285 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 43922.4us
I20260929 14:11:25.419548 5097 catalog_manager.cc:5495] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:25.425237 5086 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet a38ce5b8e5024c0ab37787aaf2b357c0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:25.447804 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet a38ce5b8e5024c0ab37787aaf2b357c0 with delete_type TABLET_DATA_TOMBSTONED (TS c5ec116e39144881bdf1f2cd3e923cf5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50768
I20260929 14:11:25.448788 6166 tablet_replica.cc:330] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5: stopping tablet replica
I20260929 14:11:25.449810 6166 raft_consensus.cc:2229] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:25.450486 6166 raft_consensus.cc:2258] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:25.453753 6166 ts_tablet_manager.cc:1905] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:25.468142 6166 ts_tablet_manager.cc:1918] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:25.468497 6166 log.cc:1196] T a38ce5b8e5024c0ab37787aaf2b357c0 P c5ec116e39144881bdf1f2cd3e923cf5: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-1-root/wals/a38ce5b8e5024c0ab37787aaf2b357c0
I20260929 14:11:25.469692 5084 catalog_manager.cc:4841] TS c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): tablet a38ce5b8e5024c0ab37787aaf2b357c0 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:26.527005 5342 consensus_queue.cc:237] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
W20260929 14:11:26.539821 5316 consensus_peers.cc:487] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 632bdccb08444a4d89ecc1cfdbab98b2. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:26.541733 5267 raft_consensus.cc:1261] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:26.542194 5197 raft_consensus.cc:1261] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:26.543378 6039 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:26.544162 6169 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:26.552501 6172 raft_consensus.cc:2932] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:26.554008 5200 raft_consensus.cc:2932] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:26.557374 5267 raft_consensus.cc:2932] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:26.571417 5350 consensus_queue.cc:237] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:26.572724 5097 catalog_manager.cc:5495] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:26.586241 5267 raft_consensus.cc:1261] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:26.587970 6172 consensus_queue.cc:1035] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:26.588972 5200 raft_consensus.cc:1261] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 14:11:26.591768 5316 consensus_peers.cc:487] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 19257f7a5d364214b38d26c5f9b6d190. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:26.592686 6172 consensus_queue.cc:1035] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:26.600152 6172 raft_consensus.cc:2932] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:26.602617 5267 raft_consensus.cc:2932] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:26.620311 5275 consensus_queue.cc:237] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:26.627949 5099 catalog_manager.cc:5495] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:26.630774 5200 raft_consensus.cc:2932] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:26.636518 5350 raft_consensus.cc:1261] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:26.639003 5197 raft_consensus.cc:1261] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:26.640568 6173 consensus_queue.cc:1035] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260929 14:11:26.641968 5242 consensus_peers.cc:487] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08b2fdc7f66741049e11f24a1866c63f. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:26.642493 6173 consensus_queue.cc:1035] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:26.651212 6142 raft_consensus.cc:2932] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:26.655197 5350 raft_consensus.cc:2932] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:26.674256 5200 raft_consensus.cc:2932] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:26.675158 5275 consensus_queue.cc:237] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:26.675268 5098 catalog_manager.cc:5495] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
W20260929 14:11:26.696281 5242 consensus_peers.cc:487] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 678eb0db533d4bea872808b6fb39b764. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:26.696569 5200 raft_consensus.cc:1261] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:26.698089 6142 consensus_queue.cc:1035] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:26.698494 5350 raft_consensus.cc:1261] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:26.699777 6187 consensus_queue.cc:1035] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:26.714107 6187 raft_consensus.cc:2932] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:26.715724 5350 raft_consensus.cc:2932] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:26.717191 5200 raft_consensus.cc:2932] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:26.732604 5099 catalog_manager.cc:5495] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:27.044127 6191 ts_tablet_manager.cc:927] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:27.046041 6191 tablet_copy_client.cc:316] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.67:42639
I20260929 14:11:27.047799 5361 tablet_copy_service.cc:140] P d886fb1eb0474d2d9f7410c4013c45e4: Received BeginTabletCopySession request for tablet 19257f7a5d364214b38d26c5f9b6d190 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:38536)
I20260929 14:11:27.048337 5361 tablet_copy_service.cc:161] P d886fb1eb0474d2d9f7410c4013c45e4: Beginning new tablet copy session on tablet 19257f7a5d364214b38d26c5f9b6d190 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:38536: session id = 675b77a0a88a4bf08134281fc0cd949e-19257f7a5d364214b38d26c5f9b6d190
I20260929 14:11:27.055508 5361 tablet_copy_source_session.cc:205] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:27.059154 6191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19257f7a5d364214b38d26c5f9b6d190. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:27.074321 6191 tablet_copy_client.cc:791] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:27.074922 6191 tablet_copy_client.cc:655] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:27.078439 6191 tablet_copy_client.cc:531] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:27.085949 6191 tablet_bootstrap.cc:492] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:27.092090 6193 ts_tablet_manager.cc:927] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:27.094151 6193 tablet_copy_client.cc:316] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.67:42639
I20260929 14:11:27.095691 5361 tablet_copy_service.cc:140] P d886fb1eb0474d2d9f7410c4013c45e4: Received BeginTabletCopySession request for tablet 632bdccb08444a4d89ecc1cfdbab98b2 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:38536)
I20260929 14:11:27.096215 5361 tablet_copy_service.cc:161] P d886fb1eb0474d2d9f7410c4013c45e4: Beginning new tablet copy session on tablet 632bdccb08444a4d89ecc1cfdbab98b2 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:38536: session id = 675b77a0a88a4bf08134281fc0cd949e-632bdccb08444a4d89ecc1cfdbab98b2
I20260929 14:11:27.104147 6195 ts_tablet_manager.cc:927] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:27.112015 5361 tablet_copy_source_session.cc:205] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:27.122221 6193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 632bdccb08444a4d89ecc1cfdbab98b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:27.127955 6195 tablet_copy_client.cc:316] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.66:43473
I20260929 14:11:27.134460 5286 tablet_copy_service.cc:140] P c5ec116e39144881bdf1f2cd3e923cf5: Received BeginTabletCopySession request for tablet 678eb0db533d4bea872808b6fb39b764 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:41626)
I20260929 14:11:27.134979 5286 tablet_copy_service.cc:161] P c5ec116e39144881bdf1f2cd3e923cf5: Beginning new tablet copy session on tablet 678eb0db533d4bea872808b6fb39b764 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:41626: session id = 675b77a0a88a4bf08134281fc0cd949e-678eb0db533d4bea872808b6fb39b764
I20260929 14:11:27.142654 6193 tablet_copy_client.cc:791] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:27.142891 5286 tablet_copy_source_session.cc:205] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:27.143304 6193 tablet_copy_client.cc:655] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:27.146680 6193 tablet_copy_client.cc:531] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:27.147869 6195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 678eb0db533d4bea872808b6fb39b764. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:27.154345 6191 tablet_bootstrap.cc:492] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:27.155301 6191 tablet_bootstrap.cc:492] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:27.156139 6191 ts_tablet_manager.cc:1397] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.070s user 0.018s sys 0.015s
I20260929 14:11:27.157724 6193 tablet_bootstrap.cc:492] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:27.158993 6191 raft_consensus.cc:348] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:27.167675 6191 raft_consensus.cc:729] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:27.168493 6198 ts_tablet_manager.cc:927] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:27.168612 6191 consensus_queue.cc:260] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:27.179998 6195 tablet_copy_client.cc:791] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:27.180994 6195 tablet_copy_client.cc:655] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:27.182252 6198 tablet_copy_client.cc:316] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.66:43473
I20260929 14:11:27.183647 6193 tablet_bootstrap.cc:492] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:27.185410 6193 tablet_bootstrap.cc:492] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:27.186076 6193 ts_tablet_manager.cc:1397] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.013s
I20260929 14:11:27.185822 5286 tablet_copy_service.cc:140] P c5ec116e39144881bdf1f2cd3e923cf5: Received BeginTabletCopySession request for tablet 08b2fdc7f66741049e11f24a1866c63f from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:41626)
I20260929 14:11:27.186619 5286 tablet_copy_service.cc:161] P c5ec116e39144881bdf1f2cd3e923cf5: Beginning new tablet copy session on tablet 08b2fdc7f66741049e11f24a1866c63f from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:41626: session id = 675b77a0a88a4bf08134281fc0cd949e-08b2fdc7f66741049e11f24a1866c63f
I20260929 14:11:27.189209 6193 raft_consensus.cc:348] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:27.190107 6193 raft_consensus.cc:729] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:27.190716 6193 consensus_queue.cc:260] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:27.185873 6195 tablet_copy_client.cc:531] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:27.208885 6191 ts_tablet_manager.cc:1428] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.052s user 0.017s sys 0.016s
I20260929 14:11:27.210572 5286 tablet_copy_source_session.cc:205] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:27.213698 6198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08b2fdc7f66741049e11f24a1866c63f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:27.216149 6195 tablet_bootstrap.cc:492] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:27.220100 5361 tablet_copy_service.cc:342] P d886fb1eb0474d2d9f7410c4013c45e4: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-19257f7a5d364214b38d26c5f9b6d190 received from {username='slave'} at 127.0.0.1:38536
I20260929 14:11:27.225135 5361 tablet_copy_service.cc:433] P d886fb1eb0474d2d9f7410c4013c45e4: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-19257f7a5d364214b38d26c5f9b6d190 on tablet 19257f7a5d364214b38d26c5f9b6d190 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:27.230566 6193 ts_tablet_manager.cc:1428] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.044s user 0.014s sys 0.012s
I20260929 14:11:27.240545 6198 tablet_copy_client.cc:791] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:27.241045 6198 tablet_copy_client.cc:655] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:27.244809 6198 tablet_copy_client.cc:531] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:27.247846 5361 tablet_copy_service.cc:342] P d886fb1eb0474d2d9f7410c4013c45e4: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-632bdccb08444a4d89ecc1cfdbab98b2 received from {username='slave'} at 127.0.0.1:38536
I20260929 14:11:27.248291 5361 tablet_copy_service.cc:433] P d886fb1eb0474d2d9f7410c4013c45e4: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-632bdccb08444a4d89ecc1cfdbab98b2 on tablet 632bdccb08444a4d89ecc1cfdbab98b2 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:27.254330 6198 tablet_bootstrap.cc:492] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:27.281239 6198 tablet_bootstrap.cc:492] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:27.282543 6198 tablet_bootstrap.cc:492] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:27.283185 6198 ts_tablet_manager.cc:1397] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.015s
I20260929 14:11:27.285982 6198 raft_consensus.cc:348] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:27.286842 6198 raft_consensus.cc:729] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:27.287431 6198 consensus_queue.cc:260] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:27.292374 6198 ts_tablet_manager.cc:1428] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.009s user 0.005s sys 0.001s
I20260929 14:11:27.293332 6195 tablet_bootstrap.cc:492] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:27.293911 5286 tablet_copy_service.cc:342] P c5ec116e39144881bdf1f2cd3e923cf5: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-08b2fdc7f66741049e11f24a1866c63f received from {username='slave'} at 127.0.0.1:41626
I20260929 14:11:27.294334 5286 tablet_copy_service.cc:433] P c5ec116e39144881bdf1f2cd3e923cf5: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-08b2fdc7f66741049e11f24a1866c63f on tablet 08b2fdc7f66741049e11f24a1866c63f with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:27.294308 6195 tablet_bootstrap.cc:492] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:27.299520 6195 ts_tablet_manager.cc:1397] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.084s user 0.021s sys 0.000s
I20260929 14:11:27.302353 6195 raft_consensus.cc:348] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:27.303247 6195 raft_consensus.cc:729] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:27.303926 6195 consensus_queue.cc:260] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:27.306378 6195 ts_tablet_manager.cc:1428] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260929 14:11:27.308053 5286 tablet_copy_service.cc:342] P c5ec116e39144881bdf1f2cd3e923cf5: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-678eb0db533d4bea872808b6fb39b764 received from {username='slave'} at 127.0.0.1:41626
I20260929 14:11:27.308396 5286 tablet_copy_service.cc:433] P c5ec116e39144881bdf1f2cd3e923cf5: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-678eb0db533d4bea872808b6fb39b764 on tablet 678eb0db533d4bea872808b6fb39b764 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:27.441267 5903 raft_consensus.cc:1203] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:27.569772 5903 raft_consensus.cc:1203] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:27.690912 5903 raft_consensus.cc:1203] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:27.736699 5903 raft_consensus.cc:1203] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:27.889173 6169 raft_consensus.cc:1050] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:27.891824 6169 consensus_queue.cc:237] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:27.898556 5903 raft_consensus.cc:1261] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:27.898675 5267 raft_consensus.cc:1261] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:27.899888 5201 raft_consensus.cc:1261] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:27.899948 6169 consensus_queue.cc:1035] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260929 14:11:27.903715 6039 consensus_queue.cc:1035] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:27.906195 6172 consensus_queue.cc:1035] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:27.919786 6039 raft_consensus.cc:2932] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:27.922389 5200 raft_consensus.cc:2932] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:27.926666 5269 raft_consensus.cc:2932] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:27.942483 5097 catalog_manager.cc:5495] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:27.944566 5903 raft_consensus.cc:2932] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:27.972962 6173 raft_consensus.cc:1050] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:27.975257 6173 consensus_queue.cc:237] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:27.982792 5903 raft_consensus.cc:1261] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:27.984560 5344 raft_consensus.cc:1261] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:27.985178 5201 raft_consensus.cc:1261] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:27.985391 6187 consensus_queue.cc:1035] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:27.989738 6173 consensus_queue.cc:1035] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:27.991948 6203 consensus_queue.cc:1035] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:28.032902 6203 raft_consensus.cc:2932] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.037818 5344 raft_consensus.cc:2932] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.053140 5200 raft_consensus.cc:2932] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.054101 5342 consensus_queue.cc:237] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:28.059960 5197 raft_consensus.cc:1261] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:28.061177 6039 consensus_queue.cc:1035] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:28.060552 5907 raft_consensus.cc:2932] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.064551 5903 raft_consensus.cc:1261] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:28.068720 6208 consensus_queue.cc:1035] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:28.074801 6208 raft_consensus.cc:2932] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.076545 5197 raft_consensus.cc:2932] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.085237 5096 catalog_manager.cc:5495] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.097335 5903 raft_consensus.cc:2932] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.106801 5083 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 19257f7a5d364214b38d26c5f9b6d190 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:28.129768 5267 consensus_queue.cc:237] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:28.135167 5350 raft_consensus.cc:1261] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:28.136786 6187 consensus_queue.cc:1035] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:11:28.138867 5907 raft_consensus.cc:1261] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:28.139040 5098 catalog_manager.cc:5495] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:28.143287 6222 consensus_queue.cc:1035] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:28.147431 6203 raft_consensus.cc:2932] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.151124 5350 raft_consensus.cc:2932] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.163488 5084 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 678eb0db533d4bea872808b6fb39b764 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:28.179916 5907 raft_consensus.cc:2932] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.180923 5097 catalog_manager.cc:5495] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:28.185662 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet 19257f7a5d364214b38d26c5f9b6d190 with delete_type TABLET_DATA_TOMBSTONED (TS c5ec116e39144881bdf1f2cd3e923cf5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50768
I20260929 14:11:28.205763 5183 tablet_service.cc:1514] Processing DeleteTablet for tablet 678eb0db533d4bea872808b6fb39b764 with delete_type TABLET_DATA_TOMBSTONED (TS ac17d3b553b74c9da39e0ab307addacd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52180
I20260929 14:11:28.214802 6172 raft_consensus.cc:1050] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:28.217332 6172 consensus_queue.cc:237] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:28.218190 6224 tablet_replica.cc:330] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd: stopping tablet replica
I20260929 14:11:28.219169 6224 raft_consensus.cc:2229] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:28.219866 6224 raft_consensus.cc:2258] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:28.223624 6223 tablet_replica.cc:330] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5: stopping tablet replica
I20260929 14:11:28.224349 6224 ts_tablet_manager.cc:1905] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:28.226202 6223 raft_consensus.cc:2229] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:28.226730 6223 raft_consensus.cc:2258] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:28.229753 6223 ts_tablet_manager.cc:1905] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:28.242563 5907 raft_consensus.cc:1261] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:28.243098 5269 raft_consensus.cc:1261] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:28.244053 6039 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:28.247592 6208 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:28.255796 5197 raft_consensus.cc:1261] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:28.256763 6224 ts_tablet_manager.cc:1918] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:28.257239 6224 log.cc:1196] T 678eb0db533d4bea872808b6fb39b764 P ac17d3b553b74c9da39e0ab307addacd: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-0-root/wals/678eb0db533d4bea872808b6fb39b764
I20260929 14:11:28.258847 5086 catalog_manager.cc:4841] TS ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): tablet 678eb0db533d4bea872808b6fb39b764 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:28.261047 6169 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:28.278714 6223 ts_tablet_manager.cc:1918] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:28.279449 6223 log.cc:1196] T 19257f7a5d364214b38d26c5f9b6d190 P c5ec116e39144881bdf1f2cd3e923cf5: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-1-root/wals/19257f7a5d364214b38d26c5f9b6d190
I20260929 14:11:28.283313 5084 catalog_manager.cc:4841] TS c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): tablet 19257f7a5d364214b38d26c5f9b6d190 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:28.285262 6169 raft_consensus.cc:2932] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.291757 5267 raft_consensus.cc:2932] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.302974 5197 raft_consensus.cc:2932] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.307430 5099 catalog_manager.cc:5495] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:28.316414 5907 raft_consensus.cc:2932] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.336390 6187 raft_consensus.cc:1050] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:28.338909 6187 consensus_queue.cc:237] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:28.346390 5351 consensus_queue.cc:237] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:28.348322 5350 raft_consensus.cc:1261] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:28.350127 5197 raft_consensus.cc:1261] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:28.350209 6187 consensus_queue.cc:1035] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:28.353600 5903 raft_consensus.cc:1261] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:28.353302 6187 consensus_queue.cc:1035] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:28.353915 5267 raft_consensus.cc:1261] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:28.355400 6208 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:28.357733 6172 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:28.365100 6208 raft_consensus.cc:2932] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.368276 5903 raft_consensus.cc:2932] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.370163 5275 raft_consensus.cc:2932] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.377619 5907 raft_consensus.cc:1261] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:28.384101 6187 consensus_queue.cc:1035] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:28.389987 5083 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 632bdccb08444a4d89ecc1cfdbab98b2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:28.397912 5099 catalog_manager.cc:5495] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.404290 6203 raft_consensus.cc:2932] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.413043 5183 tablet_service.cc:1514] Processing DeleteTablet for tablet 632bdccb08444a4d89ecc1cfdbab98b2 with delete_type TABLET_DATA_TOMBSTONED (TS ac17d3b553b74c9da39e0ab307addacd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52180
I20260929 14:11:28.414050 6224 tablet_replica.cc:330] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd: stopping tablet replica
I20260929 14:11:28.415014 6224 raft_consensus.cc:2229] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:28.415870 6224 raft_consensus.cc:2258] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:28.416358 5350 raft_consensus.cc:2932] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.430164 6224 ts_tablet_manager.cc:1905] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:28.431828 5200 raft_consensus.cc:2932] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.467096 5099 catalog_manager.cc:5495] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:28.480525 5907 raft_consensus.cc:2932] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.502509 6224 ts_tablet_manager.cc:1918] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:28.502972 6224 log.cc:1196] T 632bdccb08444a4d89ecc1cfdbab98b2 P ac17d3b553b74c9da39e0ab307addacd: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-0-root/wals/632bdccb08444a4d89ecc1cfdbab98b2
I20260929 14:11:28.504426 5086 catalog_manager.cc:4841] TS ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): tablet 632bdccb08444a4d89ecc1cfdbab98b2 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:28.512624 5275 consensus_queue.cc:237] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:28.517313 5907 raft_consensus.cc:1261] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:28.518033 5200 raft_consensus.cc:1261] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:28.518616 6173 consensus_queue.cc:1035] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:28.519886 6203 consensus_queue.cc:1035] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:28.531883 6173 raft_consensus.cc:2932] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.534515 5200 raft_consensus.cc:2932] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.540923 5084 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 08b2fdc7f66741049e11f24a1866c63f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:28.551538 5098 catalog_manager.cc:5495] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd reported cstate change: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.553040 5907 raft_consensus.cc:2932] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:28.560923 5337 tablet_service.cc:1514] Processing DeleteTablet for tablet 08b2fdc7f66741049e11f24a1866c63f with delete_type TABLET_DATA_TOMBSTONED (TS d886fb1eb0474d2d9f7410c4013c45e4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47246
I20260929 14:11:28.562631 6233 tablet_replica.cc:330] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4: stopping tablet replica
I20260929 14:11:28.563715 6233 raft_consensus.cc:2229] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:28.564419 6233 raft_consensus.cc:2258] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:28.567075 6233 ts_tablet_manager.cc:1905] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:28.580107 6233 ts_tablet_manager.cc:1918] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:28.580438 6233 log.cc:1196] T 08b2fdc7f66741049e11f24a1866c63f P d886fb1eb0474d2d9f7410c4013c45e4: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-2-root/wals/08b2fdc7f66741049e11f24a1866c63f
I20260929 14:11:28.581688 5083 catalog_manager.cc:4841] TS d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): tablet 08b2fdc7f66741049e11f24a1866c63f (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:29.757856 5269 consensus_queue.cc:237] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:29.764683 5344 raft_consensus.cc:1261] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:29.764680 5197 raft_consensus.cc:1261] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 14:11:29.765008 5242 consensus_peers.cc:487] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44e89147efa646d3b112a40f53a086a3. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:29.766515 6238 consensus_queue.cc:1035] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:29.767629 6236 consensus_queue.cc:1035] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:29.778463 6187 raft_consensus.cc:2932] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:29.780859 5200 raft_consensus.cc:2932] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:29.783128 5344 raft_consensus.cc:2932] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:29.798062 5099 catalog_manager.cc:5495] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:29.808377 5200 consensus_queue.cc:237] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:29.818181 5269 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 14:11:29.819326 5167 consensus_peers.cc:487] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c15d9fe0c3724bfbad2156748acd468e. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:29.820175 6170 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:29.820561 5344 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:29.822520 6241 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:29.838337 6234 raft_consensus.cc:2932] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:29.840036 5269 raft_consensus.cc:2932] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:29.840747 5344 raft_consensus.cc:2932] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:29.868652 5200 consensus_queue.cc:237] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:29.871385 5097 catalog_manager.cc:5495] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
W20260929 14:11:29.878425 5167 consensus_peers.cc:487] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b7924b2689dc46c98af8c4344a1e1e33. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:29.881469 5269 raft_consensus.cc:1261] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:29.882124 5350 raft_consensus.cc:1261] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:29.883757 6242 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:29.886705 6234 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:29.893524 6242 raft_consensus.cc:2932] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:29.898891 5350 raft_consensus.cc:2932] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:29.905771 5269 raft_consensus.cc:2932] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:29.909967 5197 consensus_queue.cc:237] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:29.917948 5098 catalog_manager.cc:5495] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:29.919106 5344 raft_consensus.cc:1261] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 14:11:29.920416 5167 consensus_peers.cc:487] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91e285df817143ec9947058346e02b90. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:29.922456 6204 consensus_queue.cc:1035] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:29.928722 5275 raft_consensus.cc:1261] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:29.932241 6242 consensus_queue.cc:1035] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:29.951025 5344 raft_consensus.cc:2932] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:29.949416 6242 raft_consensus.cc:2932] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:29.964401 5275 raft_consensus.cc:2932] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:29.982726 5098 catalog_manager.cc:5495] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:30.241894 6259 ts_tablet_manager.cc:927] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:30.250226 6259 tablet_copy_client.cc:316] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.66:43473
I20260929 14:11:30.253433 5286 tablet_copy_service.cc:140] P c5ec116e39144881bdf1f2cd3e923cf5: Received BeginTabletCopySession request for tablet 44e89147efa646d3b112a40f53a086a3 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:41626)
I20260929 14:11:30.253993 5286 tablet_copy_service.cc:161] P c5ec116e39144881bdf1f2cd3e923cf5: Beginning new tablet copy session on tablet 44e89147efa646d3b112a40f53a086a3 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:41626: session id = 675b77a0a88a4bf08134281fc0cd949e-44e89147efa646d3b112a40f53a086a3
I20260929 14:11:30.270409 5286 tablet_copy_source_session.cc:205] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:30.274382 6259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44e89147efa646d3b112a40f53a086a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:30.291492 6259 tablet_copy_client.cc:791] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:30.292076 6259 tablet_copy_client.cc:655] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:30.295750 6259 tablet_copy_client.cc:531] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:30.302627 6259 tablet_bootstrap.cc:492] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:30.322386 6259 tablet_bootstrap.cc:492] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:30.323253 6259 tablet_bootstrap.cc:492] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:30.323907 6259 ts_tablet_manager.cc:1397] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.010s
I20260929 14:11:30.326560 6259 raft_consensus.cc:348] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:30.327519 6259 raft_consensus.cc:729] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:30.328172 6259 consensus_queue.cc:260] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:30.331768 6259 ts_tablet_manager.cc:1428] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260929 14:11:30.333354 5286 tablet_copy_service.cc:342] P c5ec116e39144881bdf1f2cd3e923cf5: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-44e89147efa646d3b112a40f53a086a3 received from {username='slave'} at 127.0.0.1:41626
I20260929 14:11:30.333796 5286 tablet_copy_service.cc:433] P c5ec116e39144881bdf1f2cd3e923cf5: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-44e89147efa646d3b112a40f53a086a3 on tablet 44e89147efa646d3b112a40f53a086a3 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:30.371613 6259 ts_tablet_manager.cc:927] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147)
I20260929 14:11:30.374933 6259 tablet_copy_client.cc:316] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.65:45147
I20260929 14:11:30.376945 5211 tablet_copy_service.cc:140] P ac17d3b553b74c9da39e0ab307addacd: Received BeginTabletCopySession request for tablet c15d9fe0c3724bfbad2156748acd468e from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:37270)
I20260929 14:11:30.377404 5211 tablet_copy_service.cc:161] P ac17d3b553b74c9da39e0ab307addacd: Beginning new tablet copy session on tablet c15d9fe0c3724bfbad2156748acd468e from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:37270: session id = 675b77a0a88a4bf08134281fc0cd949e-c15d9fe0c3724bfbad2156748acd468e
I20260929 14:11:30.383307 5211 tablet_copy_source_session.cc:205] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:30.385957 6259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c15d9fe0c3724bfbad2156748acd468e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:30.400712 6259 tablet_copy_client.cc:791] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:30.401295 6259 tablet_copy_client.cc:655] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:30.404165 6264 ts_tablet_manager.cc:927] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147)
I20260929 14:11:30.405484 6259 tablet_copy_client.cc:531] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:30.411602 6264 tablet_copy_client.cc:316] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.65:45147
I20260929 14:11:30.412925 6259 tablet_bootstrap.cc:492] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:30.413462 5211 tablet_copy_service.cc:140] P ac17d3b553b74c9da39e0ab307addacd: Received BeginTabletCopySession request for tablet b7924b2689dc46c98af8c4344a1e1e33 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:37270)
I20260929 14:11:30.414003 5211 tablet_copy_service.cc:161] P ac17d3b553b74c9da39e0ab307addacd: Beginning new tablet copy session on tablet b7924b2689dc46c98af8c4344a1e1e33 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:37270: session id = 675b77a0a88a4bf08134281fc0cd949e-b7924b2689dc46c98af8c4344a1e1e33
I20260929 14:11:30.425333 5211 tablet_copy_source_session.cc:205] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:30.428993 6264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7924b2689dc46c98af8c4344a1e1e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:30.434553 6259 tablet_bootstrap.cc:492] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:30.435321 6259 tablet_bootstrap.cc:492] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:30.435907 6259 ts_tablet_manager.cc:1397] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.023s user 0.017s sys 0.005s
I20260929 14:11:30.438211 6259 raft_consensus.cc:348] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:30.439080 6259 raft_consensus.cc:729] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:30.439730 6259 consensus_queue.cc:260] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:30.441871 6259 ts_tablet_manager.cc:1428] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260929 14:11:30.444808 5211 tablet_copy_service.cc:342] P ac17d3b553b74c9da39e0ab307addacd: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-c15d9fe0c3724bfbad2156748acd468e received from {username='slave'} at 127.0.0.1:37270
I20260929 14:11:30.452459 5211 tablet_copy_service.cc:433] P ac17d3b553b74c9da39e0ab307addacd: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-c15d9fe0c3724bfbad2156748acd468e on tablet c15d9fe0c3724bfbad2156748acd468e with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:30.447242 6264 tablet_copy_client.cc:791] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:30.454488 6264 tablet_copy_client.cc:655] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:30.455456 6259 ts_tablet_manager.cc:927] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147)
I20260929 14:11:30.465250 6264 tablet_copy_client.cc:531] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:30.474079 6259 tablet_copy_client.cc:316] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.65:45147
I20260929 14:11:30.475247 6264 tablet_bootstrap.cc:492] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:30.476024 5211 tablet_copy_service.cc:140] P ac17d3b553b74c9da39e0ab307addacd: Received BeginTabletCopySession request for tablet 91e285df817143ec9947058346e02b90 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:37270)
I20260929 14:11:30.476608 5211 tablet_copy_service.cc:161] P ac17d3b553b74c9da39e0ab307addacd: Beginning new tablet copy session on tablet 91e285df817143ec9947058346e02b90 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:37270: session id = 675b77a0a88a4bf08134281fc0cd949e-91e285df817143ec9947058346e02b90
I20260929 14:11:30.485461 5211 tablet_copy_source_session.cc:205] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:30.504767 6259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e285df817143ec9947058346e02b90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:30.513441 6264 tablet_bootstrap.cc:492] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:30.514554 6264 tablet_bootstrap.cc:492] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:30.515295 6264 ts_tablet_manager.cc:1397] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.040s user 0.024s sys 0.006s
I20260929 14:11:30.518374 6264 raft_consensus.cc:348] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:30.519147 6264 raft_consensus.cc:729] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:30.519841 6264 consensus_queue.cc:260] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:30.522434 6264 ts_tablet_manager.cc:1428] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260929 14:11:30.527364 5211 tablet_copy_service.cc:342] P ac17d3b553b74c9da39e0ab307addacd: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-b7924b2689dc46c98af8c4344a1e1e33 received from {username='slave'} at 127.0.0.1:37270
I20260929 14:11:30.527884 5211 tablet_copy_service.cc:433] P ac17d3b553b74c9da39e0ab307addacd: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-b7924b2689dc46c98af8c4344a1e1e33 on tablet b7924b2689dc46c98af8c4344a1e1e33 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:30.547231 6259 tablet_copy_client.cc:791] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:30.547950 6259 tablet_copy_client.cc:655] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:30.553483 6259 tablet_copy_client.cc:531] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:30.563347 6259 tablet_bootstrap.cc:492] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:30.589283 6259 tablet_bootstrap.cc:492] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:30.590123 6259 tablet_bootstrap.cc:492] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:30.590703 6259 ts_tablet_manager.cc:1397] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.028s user 0.019s sys 0.005s
I20260929 14:11:30.592592 6259 raft_consensus.cc:348] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:30.593225 6259 raft_consensus.cc:729] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:30.593719 6259 consensus_queue.cc:260] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:30.595584 6259 ts_tablet_manager.cc:1428] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260929 14:11:30.597693 5211 tablet_copy_service.cc:342] P ac17d3b553b74c9da39e0ab307addacd: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-91e285df817143ec9947058346e02b90 received from {username='slave'} at 127.0.0.1:37270
I20260929 14:11:30.598098 5211 tablet_copy_service.cc:433] P ac17d3b553b74c9da39e0ab307addacd: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-91e285df817143ec9947058346e02b90 on tablet 91e285df817143ec9947058346e02b90 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:30.818035 5907 raft_consensus.cc:1203] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:30.839305 5907 raft_consensus.cc:1203] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:30.979745 5907 raft_consensus.cc:1203] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:31.054951 5907 raft_consensus.cc:1203] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:31.258055 6241 raft_consensus.cc:1050] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:31.260628 6241 consensus_queue.cc:237] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:31.277150 5269 raft_consensus.cc:1261] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:31.278445 5906 raft_consensus.cc:1261] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:31.278409 6242 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:31.279028 5350 raft_consensus.cc:1261] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:31.280252 6242 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:31.284932 6242 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:31.290472 6271 raft_consensus.cc:1050] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:31.292788 6271 consensus_queue.cc:237] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:31.294494 6234 raft_consensus.cc:2932] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.298776 5350 raft_consensus.cc:2932] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.304486 5906 raft_consensus.cc:2932] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.321007 5099 catalog_manager.cc:5495] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.322693 5344 raft_consensus.cc:1261] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:31.322796 5906 raft_consensus.cc:1261] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:31.324623 6272 consensus_queue.cc:1035] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:31.326515 6237 consensus_queue.cc:1035] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:31.332231 5200 raft_consensus.cc:1261] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:31.337374 6238 consensus_queue.cc:1035] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:31.347595 5269 raft_consensus.cc:2932] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.377805 6272 raft_consensus.cc:2932] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.382656 5906 raft_consensus.cc:2932] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.395128 5199 raft_consensus.cc:2932] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.395982 5344 raft_consensus.cc:2932] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.418802 5098 catalog_manager.cc:5495] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.424557 5200 consensus_queue.cc:237] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:31.434789 5906 raft_consensus.cc:1261] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:31.436187 5344 raft_consensus.cc:1261] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:31.436425 6234 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:31.437644 6241 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:31.444087 6242 raft_consensus.cc:2932] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.445876 5344 raft_consensus.cc:2932] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.457625 5269 consensus_queue.cc:237] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:31.462698 5086 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet b7924b2689dc46c98af8c4344a1e1e33 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:31.465440 5907 raft_consensus.cc:1261] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:31.465627 5200 raft_consensus.cc:1261] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:31.466686 5097 catalog_manager.cc:5495] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.467173 6238 consensus_queue.cc:1035] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:31.468307 6285 consensus_queue.cc:1035] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:31.469054 5906 raft_consensus.cc:2932] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.476032 6271 raft_consensus.cc:2932] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.483839 5199 raft_consensus.cc:2932] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.486208 5907 raft_consensus.cc:2932] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.493299 5084 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 44e89147efa646d3b112a40f53a086a3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:31.500679 5096 catalog_manager.cc:5495] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:31.503859 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet b7924b2689dc46c98af8c4344a1e1e33 with delete_type TABLET_DATA_TOMBSTONED (TS c5ec116e39144881bdf1f2cd3e923cf5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50768
I20260929 14:11:31.509162 6286 tablet_replica.cc:330] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5: stopping tablet replica
I20260929 14:11:31.512789 6286 raft_consensus.cc:2229] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:31.513767 6286 raft_consensus.cc:2258] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:31.518599 5337 tablet_service.cc:1514] Processing DeleteTablet for tablet 44e89147efa646d3b112a40f53a086a3 with delete_type TABLET_DATA_TOMBSTONED (TS d886fb1eb0474d2d9f7410c4013c45e4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47246
I20260929 14:11:31.518692 6286 ts_tablet_manager.cc:1905] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:31.544242 6287 tablet_replica.cc:330] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4: stopping tablet replica
I20260929 14:11:31.544687 6241 raft_consensus.cc:1050] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:31.545528 6287 raft_consensus.cc:2229] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:31.546262 6287 raft_consensus.cc:2258] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:31.547016 6241 consensus_queue.cc:237] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:31.552381 6286 ts_tablet_manager.cc:1918] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:31.552929 6286 log.cc:1196] T b7924b2689dc46c98af8c4344a1e1e33 P c5ec116e39144881bdf1f2cd3e923cf5: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-1-root/wals/b7924b2689dc46c98af8c4344a1e1e33
I20260929 14:11:31.555572 5084 catalog_manager.cc:4841] TS c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): tablet b7924b2689dc46c98af8c4344a1e1e33 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:31.558570 6287 ts_tablet_manager.cc:1905] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:31.561820 5907 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:31.562449 5344 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:31.563609 5275 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:31.563651 6241 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260929 14:11:31.565994 6242 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:31.568215 6234 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:31.587678 6242 raft_consensus.cc:2932] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.592425 5275 raft_consensus.cc:2932] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.612707 5907 raft_consensus.cc:2932] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.619009 5344 raft_consensus.cc:2932] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.631618 5097 catalog_manager.cc:5495] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.650880 6287 ts_tablet_manager.cc:1918] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:31.656543 6287 log.cc:1196] T 44e89147efa646d3b112a40f53a086a3 P d886fb1eb0474d2d9f7410c4013c45e4: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-2-root/wals/44e89147efa646d3b112a40f53a086a3
I20260929 14:11:31.664306 5200 consensus_queue.cc:237] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:31.668848 5083 catalog_manager.cc:4841] TS d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): tablet 44e89147efa646d3b112a40f53a086a3 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:31.672777 6291 raft_consensus.cc:1050] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:31.672801 5906 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:31.674643 5344 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:31.676071 6291 consensus_queue.cc:237] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:31.676970 6290 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:11:31.679661 6290 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:31.689564 5275 raft_consensus.cc:1261] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:31.690542 5351 raft_consensus.cc:1261] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:31.690953 5906 raft_consensus.cc:1261] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:31.693538 6290 consensus_queue.cc:1035] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:31.694470 6241 consensus_queue.cc:1035] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:31.700618 6295 raft_consensus.cc:2932] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.701975 5344 raft_consensus.cc:2932] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.706506 5907 raft_consensus.cc:2932] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.712667 6242 consensus_queue.cc:1035] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:31.714929 5086 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c15d9fe0c3724bfbad2156748acd468e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:31.727519 5099 catalog_manager.cc:5495] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd reported cstate change: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:31.732801 6234 raft_consensus.cc:2932] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.733667 5350 raft_consensus.cc:2932] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.743278 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet c15d9fe0c3724bfbad2156748acd468e with delete_type TABLET_DATA_TOMBSTONED (TS c5ec116e39144881bdf1f2cd3e923cf5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50768
I20260929 14:11:31.744258 6286 tablet_replica.cc:330] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5: stopping tablet replica
I20260929 14:11:31.745481 6286 raft_consensus.cc:2229] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:31.746227 6286 raft_consensus.cc:2258] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:31.750229 6286 ts_tablet_manager.cc:1905] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:31.756896 5275 raft_consensus.cc:2932] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.777987 5099 catalog_manager.cc:5495] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.777055 6286 ts_tablet_manager.cc:1918] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:31.780489 6286 log.cc:1196] T c15d9fe0c3724bfbad2156748acd468e P c5ec116e39144881bdf1f2cd3e923cf5: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-1-root/wals/c15d9fe0c3724bfbad2156748acd468e
I20260929 14:11:31.782207 5084 catalog_manager.cc:4841] TS c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): tablet c15d9fe0c3724bfbad2156748acd468e (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:31.787557 5906 raft_consensus.cc:2932] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.814256 5200 consensus_queue.cc:237] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:31.819347 5275 raft_consensus.cc:1261] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:31.819347 5906 raft_consensus.cc:1261] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:31.820674 6290 consensus_queue.cc:1035] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:31.821550 6241 consensus_queue.cc:1035] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:31.829828 6290 raft_consensus.cc:2932] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.832875 5906 raft_consensus.cc:2932] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.835013 5275 raft_consensus.cc:2932] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.838515 5086 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 91e285df817143ec9947058346e02b90 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:31.844141 5099 catalog_manager.cc:5495] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New cstate: current_term: 1 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:31.855554 5337 tablet_service.cc:1514] Processing DeleteTablet for tablet 91e285df817143ec9947058346e02b90 with delete_type TABLET_DATA_TOMBSTONED (TS d886fb1eb0474d2d9f7410c4013c45e4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47246
I20260929 14:11:31.856482 6287 tablet_replica.cc:330] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4: stopping tablet replica
I20260929 14:11:31.857465 6287 raft_consensus.cc:2229] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:31.857978 6287 raft_consensus.cc:2258] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:31.860783 6287 ts_tablet_manager.cc:1905] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:31.872148 6287 ts_tablet_manager.cc:1918] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:31.872577 6287 log.cc:1196] T 91e285df817143ec9947058346e02b90 P d886fb1eb0474d2d9f7410c4013c45e4: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-2-root/wals/91e285df817143ec9947058346e02b90
I20260929 14:11:31.874308 5083 catalog_manager.cc:4841] TS d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): tablet 91e285df817143ec9947058346e02b90 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:32.908012 5275 consensus_queue.cc:237] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:32.914215 5350 raft_consensus.cc:1261] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 14:11:32.915500 5242 consensus_peers.cc:487] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 520249ed5be343dcb30c86d3396491a4. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:32.915967 6272 consensus_queue.cc:1035] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:32.921286 5200 raft_consensus.cc:1261] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:32.925282 6301 consensus_queue.cc:1035] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:32.931092 6272 raft_consensus.cc:2932] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:32.933861 5350 raft_consensus.cc:2932] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:32.935613 5200 raft_consensus.cc:2932] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:32.948264 5096 catalog_manager.cc:5495] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:32.951858 5350 consensus_queue.cc:237] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
W20260929 14:11:32.968261 5316 consensus_peers.cc:487] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d653defbcc7e47ca89b35b1383e4e2c8. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:32.968746 5275 raft_consensus.cc:1261] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:32.970149 6288 consensus_queue.cc:1035] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:32.971182 5200 raft_consensus.cc:1261] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:32.972640 6288 consensus_queue.cc:1035] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:32.983090 6039 raft_consensus.cc:2932] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:32.984923 5275 raft_consensus.cc:2932] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:33.007395 5200 raft_consensus.cc:2932] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:33.025668 5098 catalog_manager.cc:5495] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:33.061775 5200 consensus_queue.cc:237] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:33.067999 5269 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:33.068037 5350 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260929 14:11:33.068177 5167 consensus_peers.cc:487] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d4dd0a13d8aa4e6bbd3633c047984f67. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:33.069469 6241 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:33.070267 6290 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:33.080718 6290 raft_consensus.cc:2932] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:33.083858 5350 raft_consensus.cc:2932] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:33.087024 5269 raft_consensus.cc:2932] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:33.105146 5275 consensus_queue.cc:237] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:33.108279 5098 catalog_manager.cc:5495] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 2 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
W20260929 14:11:33.126935 5242 consensus_peers.cc:487] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 -> Peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497): Couldn't send request to peer 675b77a0a88a4bf08134281fc0cd949e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f84af62485a4405f887bb8675221641e. This is attempt 1: this message will repeat every 5th retry.
I20260929 14:11:33.128284 5350 raft_consensus.cc:1261] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:33.129712 6301 consensus_queue.cc:1035] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:33.132804 5200 raft_consensus.cc:1261] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 14:11:33.134747 6301 consensus_queue.cc:1035] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:33.139868 6285 raft_consensus.cc:2932] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:33.153718 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:11:33.149437 5350 raft_consensus.cc:2932] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:33.156837 5200 raft_consensus.cc:2932] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } } }
I20260929 14:11:33.172943 5096 catalog_manager.cc:5495] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index -1 to 2, NON_VOTER 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) added. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:33.255380 5350 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:11:33.255947 5350 raft_consensus.cc:595] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:33.272018 5199 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3565847d030f4cc49b113969f438c801"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:11:33.272665 5199 raft_consensus.cc:595] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Received request to transfer leadership to TS d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:33.276612 6290 raft_consensus.cc:979] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd: : Instructing follower d886fb1eb0474d2d9f7410c4013c45e4 to start an election
I20260929 14:11:33.277097 6319 raft_consensus.cc:1067] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Signalling peer d886fb1eb0474d2d9f7410c4013c45e4 to start an election
I20260929 14:11:33.278920 5350 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3565847d030f4cc49b113969f438c801"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
from {username='slave'} at 127.0.0.1:47252
I20260929 14:11:33.279472 5350 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:33.279896 5350 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.294718 5269 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:48196
I20260929 14:11:33.294790 5350 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:33.295470 5269 raft_consensus.cc:595] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:33.298396 5197 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "675b77a0a88a4bf08134281fc0cd949e"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:11:33.298483 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:33.299008 5197 raft_consensus.cc:595] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Received request to transfer leadership to TS 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:33.299115 5269 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.297931 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:33.300558 5199 raft_consensus.cc:3032] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:33.301311 5197 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:11:33.301816 5197 raft_consensus.cc:595] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Received request to transfer leadership to TS d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:33.300948 5199 raft_consensus.cc:729] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:33.303958 5197 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "675b77a0a88a4bf08134281fc0cd949e"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:11:33.304446 5197 raft_consensus.cc:595] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Received request to transfer leadership to TS 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:33.305323 5350 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:33.306684 5197 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:11:33.306932 5269 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:33.307147 5197 raft_consensus.cc:595] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:33.308295 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260929 14:11:33.308327 5314 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:33.309149 6315 raft_consensus.cc:2781] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:33.309653 6315 raft_consensus.cc:686] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:33.309955 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:11:33.310806 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:11:33.310588 6315 consensus_queue.cc:237] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:33.306169 5199 consensus_queue.cc:260] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:33.314651 5199 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.322769 5199 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:33.343273 5098 catalog_manager.cc:5495] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 2 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:33.350145 6329 ts_tablet_manager.cc:927] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:33.352249 6329 tablet_copy_client.cc:316] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.66:43473
I20260929 14:11:33.353919 5286 tablet_copy_service.cc:140] P c5ec116e39144881bdf1f2cd3e923cf5: Received BeginTabletCopySession request for tablet 520249ed5be343dcb30c86d3396491a4 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:41626)
I20260929 14:11:33.354529 5286 tablet_copy_service.cc:161] P c5ec116e39144881bdf1f2cd3e923cf5: Beginning new tablet copy session on tablet 520249ed5be343dcb30c86d3396491a4 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:41626: session id = 675b77a0a88a4bf08134281fc0cd949e-520249ed5be343dcb30c86d3396491a4
I20260929 14:11:33.366284 5286 tablet_copy_source_session.cc:205] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:33.369282 6329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 520249ed5be343dcb30c86d3396491a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:33.383158 6329 tablet_copy_client.cc:791] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:33.383762 6329 tablet_copy_client.cc:655] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:33.387828 6329 tablet_copy_client.cc:531] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:33.395418 6329 tablet_bootstrap.cc:492] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:33.413501 6329 tablet_bootstrap.cc:492] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:33.414252 6329 tablet_bootstrap.cc:492] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:33.414858 6329 ts_tablet_manager.cc:1397] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.004s
I20260929 14:11:33.417459 6329 raft_consensus.cc:348] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:33.418149 6329 raft_consensus.cc:729] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:33.418696 6329 consensus_queue.cc:260] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:33.422472 6329 ts_tablet_manager.cc:1428] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260929 14:11:33.424341 5286 tablet_copy_service.cc:342] P c5ec116e39144881bdf1f2cd3e923cf5: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-520249ed5be343dcb30c86d3396491a4 received from {username='slave'} at 127.0.0.1:41626
I20260929 14:11:33.424682 5286 tablet_copy_service.cc:433] P c5ec116e39144881bdf1f2cd3e923cf5: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-520249ed5be343dcb30c86d3396491a4 on tablet 520249ed5be343dcb30c86d3396491a4 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:33.479122 6329 ts_tablet_manager.cc:927] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:33.480708 6329 tablet_copy_client.cc:316] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.67:42639
I20260929 14:11:33.482458 5361 tablet_copy_service.cc:140] P d886fb1eb0474d2d9f7410c4013c45e4: Received BeginTabletCopySession request for tablet d653defbcc7e47ca89b35b1383e4e2c8 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:38536)
I20260929 14:11:33.482924 5361 tablet_copy_service.cc:161] P d886fb1eb0474d2d9f7410c4013c45e4: Beginning new tablet copy session on tablet d653defbcc7e47ca89b35b1383e4e2c8 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:38536: session id = 675b77a0a88a4bf08134281fc0cd949e-d653defbcc7e47ca89b35b1383e4e2c8
I20260929 14:11:33.497704 5361 tablet_copy_source_session.cc:205] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:33.501293 6329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d653defbcc7e47ca89b35b1383e4e2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:33.517292 6329 tablet_copy_client.cc:791] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:33.517846 6329 tablet_copy_client.cc:655] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:33.521770 6329 tablet_copy_client.cc:531] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:33.522541 6319 raft_consensus.cc:979] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd: : Instructing follower c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:11:33.523116 6319 raft_consensus.cc:1067] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Signalling peer c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:11:33.524605 5275 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:50770
I20260929 14:11:33.525111 5275 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:33.525490 5275 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.530611 6329 tablet_bootstrap.cc:492] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:33.531695 5275 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:33.534240 5275 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:33.534775 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:33.535346 5199 raft_consensus.cc:3032] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:33.535591 5199 raft_consensus.cc:729] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:33.536188 5199 consensus_queue.cc:260] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:33.537333 5199 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.537441 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:33.538064 5344 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.545480 5344 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:33.546904 5239 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:33.548076 6285 raft_consensus.cc:2781] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:33.548525 6285 raft_consensus.cc:686] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:33.549566 6285 consensus_queue.cc:237] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:33.556843 6329 tablet_bootstrap.cc:492] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:33.560943 6329 tablet_bootstrap.cc:492] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:33.561282 6336 ts_tablet_manager.cc:927] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147)
I20260929 14:11:33.561626 6329 ts_tablet_manager.cc:1397] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.031s user 0.023s sys 0.004s
I20260929 14:11:33.546038 5199 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:33.564275 6329 raft_consensus.cc:348] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:33.565095 6329 raft_consensus.cc:729] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:33.565755 6329 consensus_queue.cc:260] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:33.568861 6329 ts_tablet_manager.cc:1428] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260929 14:11:33.570700 5361 tablet_copy_service.cc:342] P d886fb1eb0474d2d9f7410c4013c45e4: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-d653defbcc7e47ca89b35b1383e4e2c8 received from {username='slave'} at 127.0.0.1:38536
I20260929 14:11:33.571079 5361 tablet_copy_service.cc:433] P d886fb1eb0474d2d9f7410c4013c45e4: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-d653defbcc7e47ca89b35b1383e4e2c8 on tablet d653defbcc7e47ca89b35b1383e4e2c8 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:33.574021 5098 catalog_manager.cc:5495] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 2 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:33.585201 6336 tablet_copy_client.cc:316] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.65:45147
I20260929 14:11:33.593431 5211 tablet_copy_service.cc:140] P ac17d3b553b74c9da39e0ab307addacd: Received BeginTabletCopySession request for tablet d4dd0a13d8aa4e6bbd3633c047984f67 from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:37270)
I20260929 14:11:33.593999 5211 tablet_copy_service.cc:161] P ac17d3b553b74c9da39e0ab307addacd: Beginning new tablet copy session on tablet d4dd0a13d8aa4e6bbd3633c047984f67 from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:37270: session id = 675b77a0a88a4bf08134281fc0cd949e-d4dd0a13d8aa4e6bbd3633c047984f67
I20260929 14:11:33.602123 6313 raft_consensus.cc:979] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:11:33.602679 6313 raft_consensus.cc:1067] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Signalling peer c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:11:33.603425 5211 tablet_copy_source_session.cc:205] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:33.604698 5275 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:50776
I20260929 14:11:33.607709 5275 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:33.608106 5275 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.613488 6336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4dd0a13d8aa4e6bbd3633c047984f67. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:33.615973 6290 raft_consensus.cc:979] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd: : Instructing follower 675b77a0a88a4bf08134281fc0cd949e to start an election
I20260929 14:11:33.616463 6290 raft_consensus.cc:1067] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Signalling peer 675b77a0a88a4bf08134281fc0cd949e to start an election
I20260929 14:11:33.618352 5907 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168"
dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
from {username='slave'} at 127.0.0.1:45232
I20260929 14:11:33.618841 5907 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:33.619138 5907 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.619774 5275 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:33.622640 5275 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:33.624032 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:33.624577 5350 raft_consensus.cc:3032] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:33.624852 5350 raft_consensus.cc:729] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:33.626284 5350 consensus_queue.cc:260] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:33.626924 6329 ts_tablet_manager.cc:927] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e: Initiating tablet copy from peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:33.627501 5350 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.628325 6336 tablet_copy_client.cc:791] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:33.628837 6336 tablet_copy_client.cc:655] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:33.630977 5907 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:33.632777 5907 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:33.634203 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:33.634752 5275 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.634843 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:33.635424 5197 raft_consensus.cc:3032] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:33.635454 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:33.636171 5199 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.635778 5197 raft_consensus.cc:729] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:33.637010 6336 tablet_copy_client.cc:531] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:33.637514 5197 consensus_queue.cc:260] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:33.638706 5197 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.645169 5275 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:33.647084 6336 tablet_bootstrap.cc:492] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:33.647342 5871 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:33.648494 6332 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:33.654563 5350 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:33.645036 5197 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:33.664027 5239 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:33.665150 6301 raft_consensus.cc:2781] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:33.665764 6301 raft_consensus.cc:686] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:33.666694 6301 consensus_queue.cc:237] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:33.661284 5199 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:33.674325 6332 raft_consensus.cc:686] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:33.675853 6336 tablet_bootstrap.cc:492] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:33.675252 6332 consensus_queue.cc:237] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:33.676741 6336 tablet_bootstrap.cc:492] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:33.677810 6329 tablet_copy_client.cc:316] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.242.66:43473
I20260929 14:11:33.677802 6336 ts_tablet_manager.cc:1397] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.031s user 0.012s sys 0.015s
I20260929 14:11:33.679276 5286 tablet_copy_service.cc:140] P c5ec116e39144881bdf1f2cd3e923cf5: Received BeginTabletCopySession request for tablet f84af62485a4405f887bb8675221641e from peer 675b77a0a88a4bf08134281fc0cd949e ({username='slave'} at 127.0.0.1:41626)
I20260929 14:11:33.679849 5286 tablet_copy_service.cc:161] P c5ec116e39144881bdf1f2cd3e923cf5: Beginning new tablet copy session on tablet f84af62485a4405f887bb8675221641e from peer 675b77a0a88a4bf08134281fc0cd949e at {username='slave'} at 127.0.0.1:41626: session id = 675b77a0a88a4bf08134281fc0cd949e-f84af62485a4405f887bb8675221641e
I20260929 14:11:33.691051 6336 raft_consensus.cc:348] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:33.692087 6336 raft_consensus.cc:729] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:33.692703 6336 consensus_queue.cc:260] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:33.695592 6336 ts_tablet_manager.cc:1428] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20260929 14:11:33.699204 6320 raft_consensus.cc:979] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd: : Instructing follower d886fb1eb0474d2d9f7410c4013c45e4 to start an election
I20260929 14:11:33.700065 5211 tablet_copy_service.cc:342] P ac17d3b553b74c9da39e0ab307addacd: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-d4dd0a13d8aa4e6bbd3633c047984f67 received from {username='slave'} at 127.0.0.1:37270
I20260929 14:11:33.700533 5211 tablet_copy_service.cc:433] P ac17d3b553b74c9da39e0ab307addacd: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-d4dd0a13d8aa4e6bbd3633c047984f67 on tablet d4dd0a13d8aa4e6bbd3633c047984f67 with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:33.692652 5098 catalog_manager.cc:5495] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 2, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 2 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:33.702401 5286 tablet_copy_source_session.cc:205] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 14:11:33.703418 6320 raft_consensus.cc:1067] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Signalling peer d886fb1eb0474d2d9f7410c4013c45e4 to start an election
I20260929 14:11:33.709794 5342 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
from {username='slave'} at 127.0.0.1:47252
I20260929 14:11:33.710340 5342 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:33.710692 5342 raft_consensus.cc:3037] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.712415 6329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f84af62485a4405f887bb8675221641e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 14:11:33.718998 5096 catalog_manager.cc:5495] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 2 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:33.724104 5342 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:33.727188 5342 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:33.732177 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:33.732759 5272 raft_consensus.cc:3037] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.734089 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:33.734795 5195 raft_consensus.cc:3032] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:33.735106 5195 raft_consensus.cc:729] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:33.735939 5195 consensus_queue.cc:260] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:33.737725 5195 raft_consensus.cc:3037] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.738368 5269 raft_consensus.cc:1261] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:33.745407 6313 consensus_queue.cc:1035] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:33.749954 5195 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:33.756779 6320 raft_consensus.cc:979] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd: : Instructing follower 675b77a0a88a4bf08134281fc0cd949e to start an election
I20260929 14:11:33.757295 6320 raft_consensus.cc:1067] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Signalling peer 675b77a0a88a4bf08134281fc0cd949e to start an election
I20260929 14:11:33.754456 6329 tablet_copy_client.cc:791] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 0 data blocks...
I20260929 14:11:33.758819 6329 tablet_copy_client.cc:655] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Starting download of 1 WAL segments...
I20260929 14:11:33.758910 5907 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e"
dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
from {username='slave'} at 127.0.0.1:45232
I20260929 14:11:33.759472 5907 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:33.759800 5907 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.763180 6329 tablet_copy_client.cc:531] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 14:11:33.768994 5316 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:33.770380 6315 raft_consensus.cc:2781] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:33.770843 6315 raft_consensus.cc:686] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:33.771667 6315 consensus_queue.cc:237] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:33.773290 6329 tablet_bootstrap.cc:492] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap starting.
I20260929 14:11:33.778438 5272 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:33.790349 5907 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:33.793172 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:33.793816 5197 raft_consensus.cc:3032] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:33.794101 5197 raft_consensus.cc:729] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:33.795161 5195 raft_consensus.cc:1261] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:33.795806 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:33.796392 5350 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.796623 6315 consensus_queue.cc:1035] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:33.796834 5098 catalog_manager.cc:5495] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 2 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:33.798449 6329 tablet_bootstrap.cc:492] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 14:11:33.802977 6329 tablet_bootstrap.cc:492] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e: Bootstrap complete.
I20260929 14:11:33.794783 5197 consensus_queue.cc:260] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:33.803740 6329 ts_tablet_manager.cc:1397] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e: Time spent bootstrapping tablet: real 0.031s user 0.017s sys 0.001s
I20260929 14:11:33.803915 5197 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.806254 6329 raft_consensus.cc:348] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:33.807109 6329 raft_consensus.cc:729] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Initialized, Role: LEARNER
I20260929 14:11:33.807757 6329 consensus_queue.cc:260] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: NON_VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: true } }
I20260929 14:11:33.810009 6329 ts_tablet_manager.cc:1428] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260929 14:11:33.813205 5286 tablet_copy_service.cc:342] P c5ec116e39144881bdf1f2cd3e923cf5: Request end of tablet copy session 675b77a0a88a4bf08134281fc0cd949e-f84af62485a4405f887bb8675221641e received from {username='slave'} at 127.0.0.1:41626
I20260929 14:11:33.813623 5286 tablet_copy_service.cc:433] P c5ec116e39144881bdf1f2cd3e923cf5: ending tablet copy session 675b77a0a88a4bf08134281fc0cd949e-f84af62485a4405f887bb8675221641e on tablet f84af62485a4405f887bb8675221641e with peer 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:33.819311 5350 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:33.820447 5870 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:33.821431 6332 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:33.821990 6332 raft_consensus.cc:686] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:33.822363 5197 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:33.792640 5907 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:33.823025 6332 consensus_queue.cc:237] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:33.839581 5096 catalog_manager.cc:5495] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 2 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:33.901054 6301 raft_consensus.cc:979] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:11:33.901583 6342 raft_consensus.cc:1067] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:11:33.903112 5197 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52188
I20260929 14:11:33.903539 5197 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:33.903816 5197 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.910472 5197 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:33.912598 5197 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:33.912684 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:33.913133 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:33.913622 5269 raft_consensus.cc:3032] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:33.913731 5350 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.913969 5269 raft_consensus.cc:729] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:33.914785 5269 consensus_queue.cc:260] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:33.915985 5269 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:33.920128 5350 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:33.921489 5164 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:33.921770 5269 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:33.922289 6290 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:33.922735 6290 raft_consensus.cc:686] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:33.923568 6290 consensus_queue.cc:237] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:33.935410 5099 catalog_manager.cc:5495] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 1 to 2, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 2 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:33.947346 5907 raft_consensus.cc:1203] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260929 14:11:33.964584 5907 raft_consensus.cc:1203] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:33.999078 5195 raft_consensus.cc:1261] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:34.000223 6285 consensus_queue.cc:1035] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:34.012295 5350 raft_consensus.cc:1261] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:34.013489 6301 consensus_queue.cc:1035] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:34.022099 5907 raft_consensus.cc:1203] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:34.078758 5907 raft_consensus.cc:1203] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260929 14:11:34.118422 5197 raft_consensus.cc:1261] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:34.119655 6342 consensus_queue.cc:1035] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:34.135215 5201 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:34.136212 6332 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:11:34.156656 5269 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:34.158319 6339 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:11:34.165489 5351 raft_consensus.cc:1261] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:34.167621 6342 consensus_queue.cc:1035] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:34.221944 5197 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:34.223171 6332 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:11:34.231554 5350 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:34.233531 6332 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:11:34.298277 5272 raft_consensus.cc:1261] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:34.299510 6314 consensus_queue.cc:1035] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:34.329295 5195 raft_consensus.cc:1261] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:34.330502 6313 consensus_queue.cc:1035] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:34.347252 5269 raft_consensus.cc:1261] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:34.348870 6319 consensus_queue.cc:1035] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:34.376883 5350 raft_consensus.cc:1261] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:34.379030 6319 consensus_queue.cc:1035] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:34.382794 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:11:34.398880 6290 raft_consensus.cc:1050] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:34.403414 6290 consensus_queue.cc:237] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:34.412906 5350 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260929 14:11:34.414702 6369 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:34.417567 6342 raft_consensus.cc:1050] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:34.420045 6342 consensus_queue.cc:237] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:34.426296 5907 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEARNER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260929 14:11:34.430271 6319 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:34.441722 5272 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260929 14:11:34.443363 6290 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:34.444954 5195 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "675b77a0a88a4bf08134281fc0cd949e"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:11:34.445562 5195 raft_consensus.cc:595] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Received request to transfer leadership to TS 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:34.459715 6314 raft_consensus.cc:1050] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:34.466338 6314 consensus_queue.cc:237] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:34.467885 5195 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "675b77a0a88a4bf08134281fc0cd949e"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:11:34.468484 5195 raft_consensus.cc:595] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Received request to transfer leadership to TS 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:34.469051 6369 raft_consensus.cc:2932] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:34.471814 5342 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:11:34.472375 5342 raft_consensus.cc:595] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:34.474826 5195 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:11:34.475337 5195 raft_consensus.cc:595] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Received request to transfer leadership to TS d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:34.475864 5342 raft_consensus.cc:1261] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:34.477147 6342 consensus_queue.cc:1035] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:34.477630 5195 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91e285df817143ec9947058346e02b90"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "675b77a0a88a4bf08134281fc0cd949e"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:11:34.478186 5195 raft_consensus.cc:595] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Received request to transfer leadership to TS 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:34.476437 5907 raft_consensus.cc:2932] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:34.480545 5195 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "675b77a0a88a4bf08134281fc0cd949e"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:11:34.481057 5195 raft_consensus.cc:595] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Received request to transfer leadership to TS 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:34.481776 5906 raft_consensus.cc:1261] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:34.484536 5272 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:48196
I20260929 14:11:34.485044 5272 raft_consensus.cc:595] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:34.486300 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260929 14:11:34.487952 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:11:34.488528 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:11:34.492856 5342 raft_consensus.cc:2932] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:34.494720 5097 catalog_manager.cc:5495] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:34.498304 5195 raft_consensus.cc:1261] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:34.506224 6360 consensus_queue.cc:1035] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:34.508704 6359 consensus_queue.cc:1035] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260929 14:11:34.508704 5272 raft_consensus.cc:2932] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } attrs { replace: true } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:34.539793 6319 raft_consensus.cc:979] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd: : Instructing follower 675b77a0a88a4bf08134281fc0cd949e to start an election
I20260929 14:11:34.540302 6319 raft_consensus.cc:1067] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Signalling peer 675b77a0a88a4bf08134281fc0cd949e to start an election
I20260929 14:11:34.543061 6360 raft_consensus.cc:979] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:11:34.542955 5903 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670"
dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
from {username='slave'} at 127.0.0.1:45232
I20260929 14:11:34.543973 5903 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:34.544314 5903 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:34.543545 6360 raft_consensus.cc:1067] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:11:34.546520 5197 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52188
I20260929 14:11:34.547053 5197 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:34.547394 5197 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:34.554726 5201 consensus_queue.cc:237] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:34.562500 5906 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260929 14:11:34.563256 5267 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260929 14:11:34.563959 6290 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:34.565001 6369 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:11:34.571071 5903 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:34.576187 5197 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:34.582619 5197 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:34.584055 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:34.579933 6320 raft_consensus.cc:2932] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:34.584666 5351 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:34.585750 5903 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:34.586607 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:34.587142 5344 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:34.592000 5351 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:34.602680 5904 raft_consensus.cc:1261] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:34.605002 5164 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:34.605314 6315 consensus_queue.cc:1035] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:34.608134 6290 raft_consensus.cc:2781] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:34.615593 5272 raft_consensus.cc:1261] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:34.617883 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:34.618744 5200 raft_consensus.cc:3032] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:34.619194 5200 raft_consensus.cc:729] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:34.620091 5200 consensus_queue.cc:260] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:34.621914 5200 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:34.622970 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:34.623538 5272 raft_consensus.cc:3032] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:34.623778 5272 raft_consensus.cc:729] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:34.624500 5272 consensus_queue.cc:260] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:34.625735 5272 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:34.628702 5906 raft_consensus.cc:2932] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:34.636413 6315 consensus_queue.cc:1035] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:34.641709 6290 raft_consensus.cc:686] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:34.642499 6290 consensus_queue.cc:237] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:34.682262 6315 raft_consensus.cc:979] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:11:34.661103 5200 raft_consensus.cc:2454] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:34.708382 6313 raft_consensus.cc:1067] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:11:34.663872 6369 raft_consensus.cc:979] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd: : Instructing follower 675b77a0a88a4bf08134281fc0cd949e to start an election
I20260929 14:11:34.690879 6285 raft_consensus.cc:2932] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:34.692988 5197 raft_consensus.cc:1261] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:34.671077 5344 raft_consensus.cc:2454] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
W20260929 14:11:34.697690 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 81086.8us
I20260929 14:11:34.717048 6344 consensus_queue.cc:1035] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:34.719628 5086 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d4dd0a13d8aa4e6bbd3633c047984f67 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:34.719689 5870 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:34.720947 6364 raft_consensus.cc:2781] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:34.721428 6364 raft_consensus.cc:686] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:34.722255 6364 consensus_queue.cc:237] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:34.732180 5200 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:11:34.733711 5200 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:34.735097 5200 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:34.766978 5200 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:34.775691 5907 raft_consensus.cc:2932] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:34.744879 5272 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:34.802085 5200 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:34.804738 6290 raft_consensus.cc:1067] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Signalling peer 675b77a0a88a4bf08134281fc0cd949e to start an election
I20260929 14:11:34.907197 5098 catalog_manager.cc:5495] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 2 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:34.908929 5098 catalog_manager.cc:5495] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New cstate: current_term: 2 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:34.913772 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:34.914366 5350 raft_consensus.cc:3032] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:34.914644 5350 raft_consensus.cc:729] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:34.913280 5351 raft_consensus.cc:2932] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:34.917811 5350 consensus_queue.cc:260] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:34.918948 5350 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:34.921689 6386 raft_consensus.cc:979] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd: : Instructing follower 675b77a0a88a4bf08134281fc0cd949e to start an election
I20260929 14:11:34.922137 6386 raft_consensus.cc:1067] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Signalling peer 675b77a0a88a4bf08134281fc0cd949e to start an election
I20260929 14:11:34.933787 5906 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "91e285df817143ec9947058346e02b90"
dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
from {username='slave'} at 127.0.0.1:45232
I20260929 14:11:34.934290 5906 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:34.934621 5906 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:34.936892 6388 raft_consensus.cc:979] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd: : Instructing follower d886fb1eb0474d2d9f7410c4013c45e4 to start an election
I20260929 14:11:34.937427 6388 raft_consensus.cc:1067] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Signalling peer d886fb1eb0474d2d9f7410c4013c45e4 to start an election
I20260929 14:11:34.948208 5906 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:34.950153 5904 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932"
dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
from {username='slave'} at 127.0.0.1:45232
I20260929 14:11:34.950713 5904 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:34.951046 5904 raft_consensus.cc:3037] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:34.959291 5350 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:34.961400 5269 raft_consensus.cc:2932] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
W20260929 14:11:34.967835 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 208081us
I20260929 14:11:34.968799 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:34.969440 5267 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:34.966610 5906 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:34.970009 5197 raft_consensus.cc:2932] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:34.971974 5342 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
from {username='slave'} at 127.0.0.1:47252
I20260929 14:11:34.972478 5342 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:34.972824 5342 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:34.978099 6389 raft_consensus.cc:979] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd: : Instructing follower 675b77a0a88a4bf08134281fc0cd949e to start an election
I20260929 14:11:34.980983 5342 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:34.982676 5342 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:34.985208 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:34.985764 5272 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:34.987152 5267 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:34.991853 5099 catalog_manager.cc:5495] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 2 to 3, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 3 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:34.995927 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:34.997184 6395 raft_consensus.cc:1050] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5: attempting to promote NON_VOTER 675b77a0a88a4bf08134281fc0cd949e to VOTER
I20260929 14:11:34.997308 5275 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:35.003386 5272 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:35.002954 5271 raft_consensus.cc:2932] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:35.004891 5337 tablet_service.cc:1514] Processing DeleteTablet for tablet d4dd0a13d8aa4e6bbd3633c047984f67 with delete_type TABLET_DATA_TOMBSTONED (TS d886fb1eb0474d2d9f7410c4013c45e4 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:47246
I20260929 14:11:35.005365 5871 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:35.006280 6364 raft_consensus.cc:2781] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:35.006814 6364 raft_consensus.cc:686] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:35.005947 5097 catalog_manager.cc:5495] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:35.004417 5314 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:35.007702 6364 consensus_queue.cc:237] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:35.010510 6369 raft_consensus.cc:1067] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Signalling peer 675b77a0a88a4bf08134281fc0cd949e to start an election
W20260929 14:11:35.013731 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 100834us
I20260929 14:11:35.015298 5275 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:34.980212 5904 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:35.025132 5164 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:35.027133 5906 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0"
dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
from {username='slave'} at 127.0.0.1:45232
I20260929 14:11:35.043200 5906 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:35.043648 5906 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:35.027989 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:35.025318 6403 raft_consensus.cc:2781] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:35.045313 5200 raft_consensus.cc:3032] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:35.045756 5200 raft_consensus.cc:729] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:35.045177 5907 raft_consensus.cc:2932] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:35.046583 5200 consensus_queue.cc:260] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:35.047981 5200 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:35.027999 5201 raft_consensus.cc:2932] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:35.032255 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:35.034564 6395 consensus_queue.cc:237] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:35.039678 5272 consensus_queue.cc:237] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:35.057183 5197 raft_consensus.cc:3032] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:35.069242 5197 raft_consensus.cc:729] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:35.075551 5200 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:35.078835 5197 consensus_queue.cc:260] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:35.080114 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 55057us
I20260929 14:11:35.095989 5197 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:35.084616 5903 raft_consensus.cc:1226] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
W20260929 14:11:35.091105 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 48468us
W20260929 14:11:35.102380 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 83133.9us
I20260929 14:11:35.098417 6369 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:35.107268 5906 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:35.110332 5906 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:35.122145 5342 raft_consensus.cc:1261] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:11:35.125262 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:35.125804 5195 raft_consensus.cc:3032] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:35.126130 5195 raft_consensus.cc:729] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:35.126977 6290 consensus_queue.cc:1035] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.006s
I20260929 14:11:35.133718 5904 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:35.135372 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:35.135931 5272 raft_consensus.cc:3037] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:35.136686 5195 consensus_queue.cc:260] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:35.118546 6369 raft_consensus.cc:686] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
W20260929 14:11:35.195686 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 86596.2us
I20260929 14:11:35.199803 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:35.040311 6396 raft_consensus.cc:2932] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:35.199972 6405 tablet_replica.cc:330] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4: stopping tablet replica
I20260929 14:11:35.202917 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:35.210163 5201 raft_consensus.cc:3032] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:35.210480 5201 raft_consensus.cc:729] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:35.204046 5347 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:35.217612 5201 consensus_queue.cc:260] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:35.218837 5201 raft_consensus.cc:3037] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:35.232272 5272 raft_consensus.cc:2454] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:35.237303 5347 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:35.242688 6369 consensus_queue.cc:237] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:35.249699 5870 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:35.251008 6332 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:35.251580 6332 raft_consensus.cc:686] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:35.265277 5197 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:35.267370 5201 raft_consensus.cc:2454] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:35.270942 5347 raft_consensus.cc:1261] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:35.295933 6404 consensus_queue.cc:1035] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260929 14:11:35.252621 6332 consensus_queue.cc:237] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:35.305477 5275 raft_consensus.cc:1226] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:35.319948 5098 catalog_manager.cc:5495] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:35.290680 5871 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:35.309715 6403 raft_consensus.cc:686] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
W20260929 14:11:35.332832 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 57860.6us
I20260929 14:11:35.351989 5903 raft_consensus.cc:1261] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260929 14:11:35.390200 6306 log.cc:924] Time spent T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
I20260929 14:11:35.355079 6364 raft_consensus.cc:2781] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election won for term 2
W20260929 14:11:35.401629 6408 log.cc:924] Time spent T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20260929 14:11:35.402271 5350 raft_consensus.cc:2932] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:35.337697 6403 consensus_queue.cc:237] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:35.415220 5342 raft_consensus.cc:1261] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:35.398950 6405 raft_consensus.cc:2229] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260929 14:11:35.435997 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 351775us
W20260929 14:11:35.437779 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 106050us
I20260929 14:11:35.445503 6364 raft_consensus.cc:686] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:35.448822 6402 consensus_queue.cc:1035] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.068s
I20260929 14:11:35.468588 6364 consensus_queue.cc:237] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:35.454023 6405 raft_consensus.cc:2258] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:35.494782 5906 raft_consensus.cc:2932] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
W20260929 14:11:35.498620 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 168891us
I20260929 14:11:35.499538 5097 catalog_manager.cc:5495] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 2 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:35.500558 5197 raft_consensus.cc:1261] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:35.500554 6360 raft_consensus.cc:2932] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:35.537186 5271 raft_consensus.cc:1261] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:11:35.551468 5195 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:35.554234 6398 consensus_queue.cc:1035] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:35.564961 6405 ts_tablet_manager.cc:1905] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:35.565388 6418 consensus_queue.cc:1035] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.110s
I20260929 14:11:35.568982 5351 raft_consensus.cc:1226] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
W20260929 14:11:35.571846 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 53388.7us
I20260929 14:11:35.576046 6339 consensus_queue.cc:1035] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260929 14:11:35.608358 5195 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:35.632655 5906 raft_consensus.cc:1261] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 1 LEARNER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260929 14:11:35.759157 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 100653us
I20260929 14:11:35.768344 5084 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 520249ed5be343dcb30c86d3396491a4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260929 14:11:35.769315 6407 consensus_queue.cc:1035] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.137s
W20260929 14:11:35.824636 6373 log.cc:924] Time spent T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.095s user 0.000s sys 0.007s
I20260929 14:11:35.835328 5267 raft_consensus.cc:1261] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:35.843223 5195 raft_consensus.cc:1261] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260929 14:11:35.850630 5349 raft_consensus.cc:1261] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:35.869992 5197 raft_consensus.cc:1261] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:35.868412 6394 consensus_queue.cc:1035] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260929 14:11:35.915943 5349 raft_consensus.cc:1261] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20260929 14:11:35.962899 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 95288.5us
I20260929 14:11:35.937753 6391 consensus_queue.cc:1035] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260929 14:11:35.981675 6405 ts_tablet_manager.cc:1918] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260929 14:11:35.986287 5350 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:35.987330 6405 log.cc:1196] T d4dd0a13d8aa4e6bbd3633c047984f67 P d886fb1eb0474d2d9f7410c4013c45e4: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-2-root/wals/d4dd0a13d8aa4e6bbd3633c047984f67
I20260929 14:11:36.001461 6418 consensus_queue.cc:1035] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260929 14:11:36.020336 6405 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47246 (request call id 524) took 1015 ms. Trace:
I20260929 14:11:36.011596 6426 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
W20260929 14:11:36.041991 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 420376us
I20260929 14:11:36.054399 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:11:36.001786 6400 consensus_queue.cc:1035] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.105s
I20260929 14:11:36.020848 6405 rpcz_store.cc:276] 0929 14:11:35.004597 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:35.004711 (+ 114us) service_pool.cc:227] Handling call
0929 14:11:35.145809 (+141098us) ts_tablet_manager.cc:1145] Deleting tablet d4dd0a13d8aa4e6bbd3633c047984f67
0929 14:11:35.152195 (+ 6386us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0929 14:11:35.660530 (+508335us) tablet_metadata.cc:641] Metadata flushed
0929 14:11:35.969412 (+308882us) tablet_metadata.cc:641] Metadata flushed
0929 14:11:36.020243 (+ 50831us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41233,"tablet-delete.queue_time_us":121425,"thread_start_us":78088,"threads_started":1}
I20260929 14:11:36.096390 5083 catalog_manager.cc:4841] TS d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639): tablet d4dd0a13d8aa4e6bbd3633c047984f67 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
I20260929 14:11:36.098753 5198 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:36.099236 5098 catalog_manager.cc:5495] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 1 to 2, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 2 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
W20260929 14:11:36.157577 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 373855us
I20260929 14:11:36.205657 5269 raft_consensus.cc:1261] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:36.260344 5272 raft_consensus.cc:1261] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:36.240005 5197 raft_consensus.cc:1261] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260929 14:11:36.298765 6434 log.cc:924] Time spent T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.077s user 0.000s sys 0.001s
I20260929 14:11:36.292284 6364 consensus_queue.cc:1035] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
W20260929 14:11:36.340958 6441 log.cc:924] Time spent T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.104s user 0.001s sys 0.000s
W20260929 14:11:36.341137 6436 log.cc:924] Time spent T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.069s user 0.001s sys 0.000s
I20260929 14:11:36.342195 6442 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.143s
W20260929 14:11:36.369690 6435 log.cc:924] Time spent T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.178s user 0.001s sys 0.001s
I20260929 14:11:36.361388 6403 consensus_queue.cc:1035] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:36.445492 6406 consensus_queue.cc:1035] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
W20260929 14:11:36.466986 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 118423us
W20260929 14:11:36.493790 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 196380us
I20260929 14:11:36.551178 6390 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
W20260929 14:11:36.657114 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 368924us
W20260929 14:11:36.661970 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 101756us
I20260929 14:11:36.679512 6439 consensus_queue.cc:1035] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260929 14:11:36.637957 6390 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:36.746169 6390 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:36.750433 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:36.818627 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:36.830272 5164 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:37.087044 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 251838us
W20260929 14:11:37.097347 5165 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.039s sys 0.020s
W20260929 14:11:37.097940 5165 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.569s user 0.040s sys 0.020s
W20260929 14:11:37.107822 6463 log.cc:924] Time spent T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.132s user 0.000s sys 0.001s
I20260929 14:11:37.119160 6417 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260929 14:11:37.083937 5198 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s user 0.000s sys 0.056s
W20260929 14:11:37.166116 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 311765us
W20260929 14:11:37.209532 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 121582us
W20260929 14:11:37.161962 5198 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.585s user 0.000s sys 0.056s
W20260929 14:11:37.248167 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 82164.5us
W20260929 14:11:37.260761 5201 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.717s user 0.095s sys 0.136s
W20260929 14:11:37.266244 5201 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.717s user 0.096s sys 0.136s
I20260929 14:11:37.245985 6388 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:37.284610 6391 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:37.288074 6456 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:37.203068 6466 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:37.244819 6455 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
W20260929 14:11:37.307008 6464 log.cc:924] Time spent T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.145s user 0.000s sys 0.001s
I20260929 14:11:37.306346 6455 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:37.310997 6455 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:37.316025 6466 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:37.341442 6445 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:37.342312 6445 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:37.520982 6427 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:37.307958 6456 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:37.537626 6425 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:37.305171 6391 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:37.521659 6427 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:37.538196 6425 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:37.708768 6460 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 0.577s user 0.000s sys 0.019s
W20260929 14:11:37.709193 6460 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 0.603s user 0.000s sys 0.020s
I20260929 14:11:37.307499 6388 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:38.041626 6290 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:38.042361 6290 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:37.875991 5239 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.005s sys 0.014s
W20260929 14:11:38.061028 5239 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.771s user 0.008s sys 0.014s
W20260929 14:11:38.061339 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 771504us
W20260929 14:11:38.071945 5242 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.749s user 0.007s sys 0.004s
W20260929 14:11:38.072256 5242 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.764s user 0.007s sys 0.004s
I20260929 14:11:38.062479 5349 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47258 (request call id 1207) took 1580 ms. Trace:
W20260929 14:11:38.040038 6448 log.cc:924] Time spent T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.214s user 0.003s sys 0.007s
I20260929 14:11:37.713338 6466 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:37.855302 6459 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:37.749019 6431 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:37.720561 6320 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:38.027578 6473 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:38.005867 6425 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:37.754997 6391 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:38.154807 6290 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:38.639895 6290 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:38.548239 6400 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:38.265195 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:38.641865 6400 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:38.193707 5269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 988) took 1428 ms. Trace:
I20260929 14:11:38.571913 5198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1185) took 1229 ms. Trace:
I20260929 14:11:38.644291 6400 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:38.643180 5269 rpcz_store.cc:276] 0929 14:11:36.759002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:36.759117 (+ 115us) service_pool.cc:227] Handling call
0929 14:11:36.759474 (+ 357us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:36.835623 (+ 76149us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:36.835663 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:36.842851 (+ 7188us) log.cc:842] Serialized 26 byte log entry
0929 14:11:37.156566 (+313715us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:37.156604 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:37.156671 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:38.186895 (+1030224us) raft_consensus.cc:1671] finished
0929 14:11:38.186918 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:38.187114 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12139,"raft.queue_time_us":17413,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":10308,"thread_start_us":244126,"threads_started":1,"wal-append.queue_time_us":450345}
W20260929 14:11:38.179035 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 151233us
I20260929 14:11:38.590332 6478 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:38.645246 6425 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:38.645748 6425 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:38.287798 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:38.647797 6425 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:38.645748 6478 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:38.464345 6456 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:38.651443 6456 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:38.652195 6466 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:38.652560 6456 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:38.652832 6466 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:38.026726 6470 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:38.654129 6470 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:38.655283 6466 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:37.854094 6386 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:38.656065 6386 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:38.656987 6470 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:38.640503 6290 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:38.661795 6290 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:38.658893 6456 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:38.071736 6413 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:38.664284 6413 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:38.666774 6413 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:38.490187 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 412110us
I20260929 14:11:38.651552 6478 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:37.719624 6427 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:38.648284 6425 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:38.735030 6427 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:38.634963 5193 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "675b77a0a88a4bf08134281fc0cd949e"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:11:38.739882 6386 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:38.740324 5193 raft_consensus.cc:595] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Received request to transfer leadership to TS 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:38.740697 5193 raft_consensus.cc:603] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:11:38.703179 6388 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:38.737979 6492 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:38.738173 6400 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:38.739408 6427 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:38.755429 6488 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:38.755620 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:38.756203 5270 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:38.763350 6476 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:38.766310 6497 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:38.744428 6425 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:38.093653 5349 rpcz_store.cc:276] 0929 14:11:36.457902 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:36.458033 (+ 131us) service_pool.cc:227] Handling call
0929 14:11:36.460169 (+ 2136us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:36.460558 (+ 389us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:36.460595 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:36.461679 (+ 1084us) log.cc:842] Serialized 556 byte log entry
0929 14:11:36.954393 (+492714us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:36.954443 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:36.954536 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:38.037685 (+1083149us) raft_consensus.cc:1671] finished
0929 14:11:38.037701 (+ 16us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:38.037851 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":487595,"threads_started":1,"wal-append.queue_time_us":655829}
I20260929 14:11:38.741678 6492 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:38.756026 6488 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:38.843758 6488 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:38.026499 6474 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:38.614979 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:38.766964 6497 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:38.882714 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:38.755002 6400 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:38.884419 5270 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:38.764001 6476 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:38.702605 6473 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:38.704226 6431 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:38.643723 5198 rpcz_store.cc:276] 0929 14:11:37.341289 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:38.071593 (+730304us) service_pool.cc:227] Handling call
0929 14:11:38.192684 (+121091us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:38.266242 (+ 73558us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:38.303998 (+ 37756us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:38.414569 (+110571us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:38.489332 (+ 74763us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:38.523604 (+ 34272us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:38.570180 (+ 46576us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:38.772502 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:38.883394 6459 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:38.878305 6320 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:38.982882 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:38.986114 5313 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.723s user 0.004s sys 0.007s
W20260929 14:11:38.986614 5313 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.920s user 0.004s sys 0.008s
I20260929 14:11:38.912642 6492 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:38.882296 6493 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:38.989686 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:39.052592 5270 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:38.908753 5192 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 1.
W20260929 14:11:39.001974 5166 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.015s sys 0.056s
I20260929 14:11:39.001528 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:39.086426 5166 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.896s user 0.020s sys 0.061s
I20260929 14:11:39.049297 6427 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:39.081920 5165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.926s (SENT) [suppressed 4 similar messages]
I20260929 14:11:39.002436 6315 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:11:39.093223 5873 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.847s user 0.000s sys 0.022s
W20260929 14:11:39.093664 5314 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.595s user 0.000s sys 0.022s
W20260929 14:11:39.096663 5314 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.598s user 0.000s sys 0.023s
I20260929 14:11:39.097514 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:39.098222 5270 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
W20260929 14:11:39.097783 5873 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.851s user 0.000s sys 0.023s
I20260929 14:11:38.994010 6488 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:39.078035 6492 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.088936 6344 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:39.096513 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:38.891055 6474 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:39.088109 5268 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:39.001681 6490 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:11:39.111346 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 465316us
I20260929 14:11:39.001533 6485 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:39.098991 6427 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:39.122083 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:39.123346 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:39.106159 6320 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:39.124739 6290 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:39.105819 6473 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:39.129616 6459 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:39.130650 6290 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:39.115249 6425 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:39.125623 6388 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:39.134495 6476 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:39.125802 6386 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:11:39.136813 5167 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.889s user 0.033s sys 0.040s
I20260929 14:11:39.118654 6478 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:11:39.138052 5167 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.890s user 0.034s sys 0.040s
I20260929 14:11:39.111001 6492 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:39.109576 6488 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:39.157045 6484 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
W20260929 14:11:39.170269 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 47015.5us
I20260929 14:11:39.171799 6290 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:39.104285 6315 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:39.113400 6344 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:39.179761 6486 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:11:39.179318 5872 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.569s user 0.000s sys 0.018s
I20260929 14:11:39.144392 6332 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.126854 6427 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:39.114768 6490 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:39.169790 6476 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.052934 6493 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:39.135614 6388 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:39.180446 6486 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:39.143666 6386 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:39.137902 6485 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:39.321748 5201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1186) took 1223 ms. Trace:
W20260929 14:11:39.244577 5239 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.718s user 0.018s sys 0.032s
W20260929 14:11:39.192309 5315 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.590s user 0.013s sys 0.041s
W20260929 14:11:39.639029 5313 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.009s sys 0.017s
I20260929 14:11:39.478276 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:39.699854 5165 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.002s sys 0.025s
W20260929 14:11:39.775164 5239 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.272s user 0.038s sys 0.076s
W20260929 14:11:39.775475 5313 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.787s user 0.012s sys 0.022s
I20260929 14:11:39.536618 5098 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52726 (request call id 223) took 3697 ms. Trace:
W20260929 14:11:39.776068 5165 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.668s user 0.002s sys 0.029s
W20260929 14:11:39.776999 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 695090us
I20260929 14:11:39.336198 6396 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.778106 6396 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:39.779330 6493 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:39.780208 6396 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:39.780601 6485 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:39.625512 5870 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.026s user 0.029s sys 0.061s
W20260929 14:11:39.242671 6470 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.503s user 0.014s sys 0.040s
W20260929 14:11:39.777374 5872 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.432s user 0.011s sys 0.034s
I20260929 14:11:39.774447 5201 rpcz_store.cc:276] 0929 14:11:38.038267 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:38.077372 (+ 39105us) service_pool.cc:227] Handling call
0929 14:11:38.077648 (+ 276us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:38.657201 (+579553us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:38.657273 (+ 72us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:38.657332 (+ 59us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:38.657372 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:39.261297 (+603925us) spinlock_profiling.cc:243] Waited 579 ms on lock 0x7b54001446d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:39.261357 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:39.261516 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1332447488}
I20260929 14:11:39.779834 6332 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:39.780490 6388 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:39.500993 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1189) took 1404 ms. Trace:
I20260929 14:11:39.470254 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50770 (request call id 1470) took 1359 ms. Trace:
W20260929 14:11:39.702775 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 409027us
W20260929 14:11:39.781250 5870 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.182s user 0.032s sys 0.065s
W20260929 14:11:39.791117 5315 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.293s user 0.028s sys 0.058s
W20260929 14:11:39.786185 6470 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.046s user 0.030s sys 0.088s
I20260929 14:11:39.728871 6456 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.781100 6476 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:39.193058 6479 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:39.560158 6453 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:39.437949 6505 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:39.394331 6445 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:39.498684 6315 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:39.486011 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:39.215579 6497 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:39.442720 6483 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:39.864869 5194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37270 (request call id 56) took 1266 ms. Trace:
I20260929 14:11:39.864920 6483 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:39.865371 5194 rpcz_store.cc:276] 0929 14:11:38.598269 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:38.628228 (+ 29959us) service_pool.cc:227] Handling call
0929 14:11:39.485687 (+857459us) spinlock_profiling.cc:243] Waited 4.36 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:11:39.864799 (+379112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10033408}
W20260929 14:11:39.258028 6465 log.cc:924] Time spent T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 2.011s user 0.001s sys 0.027s
W20260929 14:11:39.206030 5871 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.561s user 0.023s sys 0.073s
I20260929 14:11:39.789625 5197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52188 (request call id 1193) took 1135 ms. Trace:
I20260929 14:11:39.869257 5197 rpcz_store.cc:276] 0929 14:11:38.646465 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:38.727081 (+ 80616us) service_pool.cc:227] Handling call
0929 14:11:39.478000 (+750919us) spinlock_profiling.cc:243] Waited 4.59 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:11:39.782111 (+304111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10548480}
I20260929 14:11:39.533119 6410 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:11:39.185007 5196 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.500s user 0.000s sys 0.052s
I20260929 14:11:39.874675 5200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 50) took 3362 ms. Trace:
I20260929 14:11:39.204051 6484 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:39.576639 6409 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.669180 6502 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
W20260929 14:11:39.434471 5273 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.559s user 0.020s sys 0.085s
I20260929 14:11:39.568588 6411 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.579207 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:39.824056 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 50721.7us
I20260929 14:11:39.181093 6344 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:39.377061 6461 log.cc:924] Time spent T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 2.245s user 0.006s sys 0.026s
I20260929 14:11:39.256521 6492 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:39.260895 6488 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:11:39.506853 6475 log.cc:924] Time spent T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 2.167s user 0.018s sys 0.036s
I20260929 14:11:39.437794 6400 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:39.753012 6508 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:39.206544 6431 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:39.348249 6489 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.460870 6403 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:39.857632 6456 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:39.381202 6406 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:39.438942 6474 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:39.555359 6472 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:39.382130 6469 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:39.439105 6416 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:39.429426 6499 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.825753 6394 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.859349 6490 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:39.859496 6388 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.859956 6515 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.257668 6399 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:39.820509 6485 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.866871 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:39.859364 6386 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:39.829126 6389 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.186179 6478 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:39.823244 6493 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.871217 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:39.859591 6453 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:39.873723 6410 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:39.855613 6395 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:39.593374 5099 catalog_manager.cc:5495] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index 3 to 4, VOTER ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) evicted. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:39.245239 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:39.873485 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:39.871465 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:39.834910 5199 rpcz_store.cc:276] 0929 14:11:38.097028 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:38.097146 (+ 118us) service_pool.cc:227] Handling call
0929 14:11:38.097487 (+ 341us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:38.931399 (+833912us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:38.931438 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:38.931484 (+ 46us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:38.931523 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:39.500793 (+569270us) spinlock_profiling.cc:243] Waited 834 ms on lock 0x7b54002cdb54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:39.500831 (+ 38us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:39.500939 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1917363584}
I20260929 14:11:39.831533 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:39.851007 5271 rpcz_store.cc:276] 0929 14:11:38.110886 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:38.111002 (+ 116us) service_pool.cc:227] Handling call
0929 14:11:38.636273 (+525271us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:38.669267 (+ 32994us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:38.669319 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:38.669372 (+ 53us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:38.669412 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:39.469918 (+800506us) spinlock_profiling.cc:243] Waited 32.8 ms on lock 0x7b5400392e54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:39.469982 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:39.470171 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":75341184}
I20260929 14:11:39.852200 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:39.873485 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:39.872359 6315 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:40.439121 6406 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.439931 6403 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.413779 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:40.441982 5349 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:39.879590 6411 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.443157 6386 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.879102 6409 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.444626 6411 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:39.976940 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:39.879415 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:40.439872 6315 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.443292 6406 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:39.878886 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:39.880152 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:39.932142 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:40.685932 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:40.686591 6403 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:40.687418 6410 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:11:39.871811 5871 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.227s user 0.036s sys 0.099s
I20260929 14:11:40.444998 6388 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.693758 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:39.872114 6396 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:40.391031 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:40.333859 5196 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.754s user 0.011s sys 0.073s
I20260929 14:11:40.397241 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:40.363451 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:40.446071 5192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52182 (request call id 1049) took 1321 ms. Trace:
I20260929 14:11:39.910279 6397 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:40.362470 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:40.694851 6409 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:40.713358 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:40.411171 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:40.363778 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:40.405622 6360 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:40.357722 6523 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:40.396304 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:11:39.879359 5273 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.300s user 0.023s sys 0.086s
I20260929 14:11:40.444038 6386 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:39.907676 6496 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.025s sys 0.057s
I20260929 14:11:39.875056 6510 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:39.860278 6486 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:39.995106 6483 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:40.750591 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 57716.4us
I20260929 14:11:40.751531 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:40.724571 5272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 41) took 3812 ms. Trace:
I20260929 14:11:40.725078 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:40.053719 5870 connection.cc:232] RPC call timeout handler was delayed by -1.2337901969999621s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
W20260929 14:11:40.726269 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 327002us
I20260929 14:11:40.057538 6446 raft_consensus.cc:2932] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:40.063287 6332 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:39.866418 6413 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.993534 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:39.931452 6449 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:39.879528 6407 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:40.729172 6388 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:40.726306 6397 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.727034 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:39.963929 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 110277us
I20260929 14:11:40.689055 6476 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:40.358472 6520 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:39.879940 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:40.728896 5907 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:40.749960 6396 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.061956 6484 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:39.880347 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:39.867341 6427 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:39.931950 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:40.766170 6395 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.767474 6515 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.768343 6456 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:40.712922 5192 rpcz_store.cc:276] 0929 14:11:39.124790 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:39.212525 (+ 87735us) service_pool.cc:227] Handling call
0929 14:11:40.445811 (+1233286us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:39.879711 6497 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:40.782315 6469 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:40.790081 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 321052us
I20260929 14:11:39.978523 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:40.777675 5267 raft_consensus.cc:2454] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:40.724676 6521 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:40.794850 5201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52182 (request call id 1051) took 1160 ms. Trace:
I20260929 14:11:40.795406 5201 rpcz_store.cc:276] 0929 14:11:39.583893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:39.831225 (+247332us) service_pool.cc:227] Handling call
0929 14:11:40.738263 (+907038us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:40.757622 6409 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:40.782125 6485 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:39.872344 6479 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.793917 6497 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.794090 6413 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.804113 5276 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:40.789323 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:40.768921 6416 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.752887 5272 rpcz_store.cc:276] 0929 14:11:36.912038 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:37.089256 (+177218us) service_pool.cc:227] Handling call
0929 14:11:37.089703 (+ 447us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:37.089997 (+ 294us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:37.090033 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:37.107297 (+ 17264us) log.cc:842] Serialized 26 byte log entry
0929 14:11:37.265305 (+158008us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:11:37.265350 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:37.265636 (+ 286us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:39.506341 (+2240705us) spinlock_profiling.cc:243] Waited 976 us on lock 0x7b2400132b48. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda199dcf4 00007fbda199bdfb 00007fbda1a2a559 00007fbda1a2a1e5 00007fbda1a2982d 00007fbda46d846e 00007fbda46efc0c 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551
0929 14:11:40.710732 (+1204391us) raft_consensus.cc:1671] finished
0929 14:11:40.714730 (+ 3998us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:40.724333 (+ 9603us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2245376,"thread_start_us":21691,"threads_started":1,"wal-append.queue_time_us":185365}
I20260929 14:11:39.875064 5200 rpcz_store.cc:276] 0929 14:11:36.512004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:36.513151 (+ 1147us) service_pool.cc:227] Handling call
0929 14:11:36.513561 (+ 410us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:36.513971 (+ 410us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:36.514029 (+ 58us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:36.521923 (+ 7894us) log.cc:842] Serialized 26 byte log entry
0929 14:11:36.955052 (+433129us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:11:36.955104 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:36.955185 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:38.041309 (+1086124us) spinlock_profiling.cc:243] Waited 2.27 ms on lock 0x7b48000432d8. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda99147f1 00007fbda990b8e7 00007fbda990b77a 00007fbda99ff95f 00007fbda99ff627 00007fbda9a13b2c 00007fbda9a13ad8 00007fbda9a13a58 00007fbda9a13a13 00007fbda9a12cbc 00007fbda8994be6
0929 14:11:39.257300 (+1215991us) spinlock_profiling.cc:243] Waited 2.8 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda46efbee 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551 00007fbda46f8509 00007fbda46f8499
0929 14:11:39.874466 (+617166us) raft_consensus.cc:1671] finished
0929 14:11:39.874486 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:39.874599 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":111,"raft.run_cpu_time_us":49633,"raft.run_wall_time_us":1527446,"spinlock_wait_cycles":11656320,"thread_start_us":412875,"threads_started":1,"wal-append.queue_time_us":606708}
I20260929 14:11:40.795354 6521 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:39.881568 6505 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.820029 6485 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:40.821360 6505 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:40.802109 5268 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:40.802783 6484 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:40.787142 5271 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:48196
I20260929 14:11:40.802768 6508 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.785238 6449 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:40.832647 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 69482.5us
W20260929 14:11:40.827359 6496 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.425s user 0.034s sys 0.126s
I20260929 14:11:40.837760 5346 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:40.750022 6510 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.823305 6386 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:40.799906 6360 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:39.879189 6502 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.833379 5271 raft_consensus.cc:595] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:40.857746 5270 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:40.863772 5269 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:40.798909 6523 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.831800 6411 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:40.831286 6409 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.846402 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:40.815697 6394 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.822901 6489 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.866966 6315 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:40.883720 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:40.833747 6520 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.888091 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:40.885531 5342 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:40.859822 6397 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:40.881808 6314 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:40.888953 5344 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:40.888860 5906 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:40.868474 6332 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:40.859273 6407 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:39.776585 5098 rpcz_store.cc:276] 0929 14:11:35.838980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:35.840778 (+ 1798us) service_pool.cc:227] Handling call
0929 14:11:39.536520 (+3695742us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:11:36.448668 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:11:36.449241 (+ 573us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:11:36.449280 (+ 39us) write_op.cc:435] Acquired schema lock
0929 14:11:36.449299 (+ 19us) tablet.cc:660] Decoding operations
0929 14:11:36.449664 (+ 365us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:11:36.449753 (+ 89us) write_op.cc:641] Partition lock acquired for write op
0929 14:11:36.449786 (+ 33us) tablet.cc:683] Acquiring locks for 1 operations
0929 14:11:36.450008 (+ 222us) tablet.cc:699] Row locks acquired
0929 14:11:36.450031 (+ 23us) write_op.cc:260] PREPARE: finished
0929 14:11:36.450160 (+ 129us) write_op.cc:270] Start()
0929 14:11:36.450280 (+ 120us) write_op.cc:276] Timestamp: P: 1790691096450122 usec, L: 0
0929 14:11:36.450304 (+ 24us) op_driver.cc:331] REPLICATION: starting
0929 14:11:36.450944 (+ 640us) log.cc:842] Serialized 538 byte log entry
0929 14:11:37.063233 (+612289us) op_driver.cc:444] REPLICATION: finished
0929 14:11:37.342961 (+279728us) spinlock_profiling.cc:243] Waited 1.81 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda1b275fe 00007fbdaa95ea39 00007fbda1b448bb 00007fbda1b42cbf 00007fbda1b44e22 00007fbda847d410
0929 14:11:38.255635 (+912674us) write_op.cc:301] APPLY: starting
0929 14:11:38.301459 (+ 45824us) tablet.cc:1365] starting BulkCheckPresence
0929 14:11:38.488911 (+187452us) tablet.cc:1368] finished BulkCheckPresence
0929 14:11:38.498878 (+ 9967us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:11:38.579330 (+ 80452us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:11:38.597885 (+ 18555us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0929 14:11:38.614496 (+ 16611us) write_op.cc:312] APPLY: finished
0929 14:11:38.628627 (+ 14131us) log.cc:842] Serialized 23 byte log entry
0929 14:11:38.667892 (+ 39265us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:38.725937 (+ 58045us) write_op.cc:454] Released schema lock
0929 14:11:38.730124 (+ 4187us) write_op.cc:341] FINISH: Updating metrics
0929 14:11:38.742327 (+ 12203us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:38.744905 (+ 2578us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":1050741,"apply.run_cpu_time_us":3153,"apply.run_wall_time_us":562260,"num_ops":1,"prepare.queue_time_us":331627,"prepare.run_cpu_time_us":2537,"prepare.run_wall_time_us":2538,"replication_time_us":523478,"spinlock_wait_cycles":4152576,"thread_start_us":926910,"threads_started":2,"wal-append.queue_time_us":720968,"wal-append.run_cpu_time_us":1028,"wal-append.run_wall_time_us":1040323}]]}
I20260929 14:11:40.883988 6453 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:40.911538 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:40.885450 6490 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:40.910475 5906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45248 (request call id 176) took 1033 ms. Trace:
I20260929 14:11:40.903810 5197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1054) took 1070 ms. Trace:
I20260929 14:11:40.916620 5201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1052) took 1095 ms. Trace:
I20260929 14:11:40.917146 5201 rpcz_store.cc:276] 0929 14:11:39.821377 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:40.812630 (+991253us) service_pool.cc:227] Handling call
0929 14:11:40.850143 (+ 37513us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:40.912484 (+ 62341us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:40.912519 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:40.916359 (+ 3840us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:40.916398 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:40.916457 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:40.916556 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:40.915383 5906 rpcz_store.cc:276] 0929 14:11:39.876650 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:39.876790 (+ 140us) service_pool.cc:227] Handling call
0929 14:11:40.902351 (+1025561us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:40.916285 5197 rpcz_store.cc:276] 0929 14:11:39.828595 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:40.799254 (+970659us) service_pool.cc:227] Handling call
0929 14:11:40.862642 (+ 63388us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:40.898865 (+ 36223us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:40.898904 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:40.898961 (+ 57us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:11:40.899002 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:40.899078 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:40.899206 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:40.919113 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37270 (request call id 57) took 1017 ms. Trace:
I20260929 14:11:40.881834 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:40.925484 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:40.929718 5344 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47252 (request call id 1492) took 1056 ms. Trace:
I20260929 14:11:40.889112 6484 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.932753 5344 rpcz_store.cc:276] 0929 14:11:39.866190 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:39.866912 (+ 722us) service_pool.cc:227] Handling call
0929 14:11:40.919089 (+1052177us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:40.903712 5901 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:40.938778 6359 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:40.882020 6389 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.940856 6413 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:40.941064 6469 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:40.879820 6493 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.911144 6411 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.924247 5193 rpcz_store.cc:276] 0929 14:11:39.826308 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:39.880133 (+ 53825us) service_pool.cc:227] Handling call
0929 14:11:40.839891 (+959758us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:40.947971 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:40.879171 6472 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.929755 6386 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:40.922116 6314 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.935490 6396 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:40.960079 6386 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.942081 6521 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:40.936070 5194 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:40.966359 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:40.964543 5906 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:40.967751 6391 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:40.968220 6515 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:40.968415 5195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1035) took 4475 ms. Trace:
I20260929 14:11:40.970046 6484 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:40.968850 5195 rpcz_store.cc:276] 0929 14:11:36.472473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:36.472987 (+ 514us) service_pool.cc:227] Handling call
0929 14:11:36.473611 (+ 624us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:36.496633 (+ 23022us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:36.496677 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:36.512344 (+ 15667us) log.cc:842] Serialized 26 byte log entry
0929 14:11:36.754524 (+242180us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:36.754574 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:36.754664 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:39.356684 (+2602020us) spinlock_profiling.cc:243] Waited 5 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda46efbee 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551 00007fbda46f8509 00007fbda46f8499
0929 14:11:40.914258 (+1557574us) raft_consensus.cc:1671] finished
0929 14:11:40.919114 (+ 4856us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:40.947885 (+ 28771us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22815,"raft.queue_time_us":26267,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":9718,"spinlock_wait_cycles":11497216,"thread_start_us":219575,"threads_started":1,"wal-append.queue_time_us":543830}
I20260929 14:11:40.880131 6399 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.884496 6499 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.972421 6505 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:40.973058 5194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52182 (request call id 1057) took 1116 ms. Trace:
I20260929 14:11:40.970753 6493 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:40.974773 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:40.949241 6427 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:40.964684 6489 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:40.943800 6332 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.973274 6505 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.975389 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 52) took 2777 ms. Trace:
I20260929 14:11:40.987527 6359 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.995337 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:40.989836 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:40.991209 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:40.989951 6520 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:40.973577 5194 rpcz_store.cc:276] 0929 14:11:39.856559 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:40.806498 (+949939us) service_pool.cc:227] Handling call
0929 14:11:40.972964 (+166466us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:41.000103 6314 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:40.984697 6497 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.000871 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:40.968418 6391 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:40.985141 6409 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:40.948935 6490 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:40.983765 6502 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:40.993407 6360 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:40.993451 6395 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:40.983932 6499 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:41.009737 5349 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:40.984553 6510 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.011900 6495 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:41.013990 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.014099 6399 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:41.015308 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.017510 6472 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:41.019752 5271 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:41.006971 6411 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:41.024019 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.022693 6479 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.011914 6386 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:41.000149 6416 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:41.019120 6478 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.033424 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.034526 6389 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:41.034674 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.035197 6449 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:41.010268 6427 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:41.035797 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.036844 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.037411 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.037611 6427 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.037861 6505 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:41.037926 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.038712 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.019526 5343 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:41.040413 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.040854 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.041123 6478 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:41.041292 6394 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.042070 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.041898 6478 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:41.042309 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.043293 5900 raft_consensus.cc:2454] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:41.044334 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.044428 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.045145 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.045768 5349 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:41.046675 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.026722 5314 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:41.047288 5349 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:41.040467 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.047124 6407 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:41.050179 6359 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:41.050287 5316 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:41.051386 6490 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.052268 5314 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:41.053092 6411 raft_consensus.cc:2726] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:41.029336 6495 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:41.054209 6496 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:41.053143 6472 raft_consensus.cc:2781] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:41.067960 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.068706 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.066960 5344 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:41.044454 6478 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.068702 6469 raft_consensus.cc:2781] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260929 14:11:41.070564 5871 connection.cc:232] RPC call timeout handler was delayed by -1.4355253789999551s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15580000000000002s for a response
I20260929 14:11:41.072335 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.072675 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.096858 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.097714 5350 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 1488) took 1232 ms. Trace:
I20260929 14:11:41.097740 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.098202 5350 rpcz_store.cc:276] 0929 14:11:39.864814 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:39.866390 (+ 1576us) service_pool.cc:227] Handling call
0929 14:11:39.866638 (+ 248us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:40.794405 (+927767us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:40.794449 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:40.794508 (+ 59us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:11:40.794539 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:41.097394 (+302855us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x7b540027e554. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:41.097432 (+ 38us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:41.097531 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2057451776}
I20260929 14:11:41.101518 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.101449 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.102957 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.103582 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.104018 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:41.096794 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 159897us
I20260929 14:11:41.098552 5349 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:41.106554 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.106935 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.107410 5201 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:41.108107 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.108260 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.108657 5349 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:41.109684 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.099766 5198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1047) took 1978 ms. Trace:
I20260929 14:11:41.110683 5193 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:41.110569 5198 rpcz_store.cc:276] 0929 14:11:39.121300 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:39.488697 (+367397us) service_pool.cc:227] Handling call
0929 14:11:39.489005 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:40.822304 (+1333299us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:40.822352 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:40.822408 (+ 56us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:11:40.822446 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:41.099370 (+276924us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400352854. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:41.099411 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:41.099562 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3065243392}
I20260929 14:11:41.042814 5343 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:41.024811 5273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50770 (request call id 1469) took 2861 ms. Trace:
I20260929 14:11:41.073207 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.076273 6427 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:41.076781 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.077395 5269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50776 (request call id 1004) took 1223 ms. Trace:
I20260929 14:11:41.078111 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.080322 6472 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:41.081724 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.082050 6495 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:41.082198 6530 log.cc:924] Time spent T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260929 14:11:41.082751 6508 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.084906 5192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1053) took 1261 ms. Trace:
I20260929 14:11:41.085934 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.086285 5314 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:41.086802 5167 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:41.088497 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.088753 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.092456 6469 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:41.105139 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.108268 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.092979 5164 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:41.112231 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.116804 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.117153 6523 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.138741 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.184192 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.122777 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.140688 6427 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:41.184042 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.120455 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 58) took 1153 ms. Trace:
I20260929 14:11:41.117707 6496 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:41.122254 5269 rpcz_store.cc:276] 0929 14:11:39.851761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:39.851905 (+ 144us) service_pool.cc:227] Handling call
0929 14:11:41.074568 (+1222663us) spinlock_profiling.cc:243] Waited 166 ms on lock 0x7b54001086d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:41.074705 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":381776128}
I20260929 14:11:41.318174 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.312443 5350 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
W20260929 14:11:41.192498 6535 log.cc:924] Time spent T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
W20260929 14:11:41.190075 6532 log.cc:924] Time spent T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.206s user 0.001s sys 0.000s
I20260929 14:11:41.188153 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.190116 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:40.996197 5196 rpcz_store.cc:276] 0929 14:11:38.190223 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:38.192285 (+ 2062us) service_pool.cc:227] Handling call
0929 14:11:38.303541 (+111256us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:38.497987 (+194446us) raft_consensus.cc:1520] Early marking committed up to index 5
0929 14:11:40.783884 (+2285897us) log.cc:842] Serialized 14 byte log entry
0929 14:11:40.910204 (+126320us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:40.916846 (+ 6642us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:11:40.935639 (+ 18793us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:40.948400 (+ 12761us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:40.967359 (+ 18959us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":678968,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":57,"raft.queue_time_us":2055,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":10027,"thread_start_us":2189800,"threads_started":2,"wal-append.queue_time_us":124822}
I20260929 14:11:41.198822 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:41.323217 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 237029us
I20260929 14:11:41.319468 6469 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:41.342408 5314 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:41.343904 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.338915 5201 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
W20260929 14:11:41.337769 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 263818us
I20260929 14:11:41.340348 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:11:41.184490 5346 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47252 (request call id 1499) took 1230 ms (client timeout 1639 ms). Trace:
W20260929 14:11:41.337749 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 244866us
I20260929 14:11:41.184592 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.184383 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.341867 5197 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:41.224589 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.346755 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.320307 6472 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:41.348248 5198 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:41.318645 5199 rpcz_store.cc:276] 0929 14:11:39.853899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:40.827078 (+973179us) service_pool.cc:227] Handling call
0929 14:11:40.857781 (+ 30703us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:40.875701 (+ 17920us) raft_consensus.cc:1520] Early marking committed up to index 5
0929 14:11:40.943928 (+ 68227us) log.cc:842] Serialized 14 byte log entry
0929 14:11:41.003333 (+ 59405us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:41.003366 (+ 33us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:11:41.003389 (+ 23us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:41.003429 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:41.007630 (+ 4201us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":74237,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":1901,"raft.queue_time_us":10391,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":2679,"thread_start_us":41732,"threads_started":2,"wal-append.queue_time_us":102037}
I20260929 14:11:41.349404 6490 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:41.345930 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.350029 6490 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:41.350756 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.351555 5347 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:41.352500 6490 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.353089 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.354521 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:41.345758 5346 rpcz_store.cc:269] 0929 14:11:39.954273 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:40.410345 (+456072us) service_pool.cc:227] Handling call
0929 14:11:41.156312 (+745967us) spinlock_profiling.cc:243] Waited 121 ms on lock 0x7b54002d1ed4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:41.184432 (+ 28120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":278809856}
I20260929 14:11:41.357864 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.362571 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.342077 6523 raft_consensus.cc:2781] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:41.328657 6495 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:41.367038 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.341100 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.364723 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.356025 6416 raft_consensus.cc:2781] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:41.372028 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.370752 6514 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:41.375181 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.378387 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.375715 5164 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:41.361794 6476 raft_consensus.cc:2781] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:41.380728 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.378949 5903 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:41.378021 5195 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:41.373698 6510 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:41.367177 6332 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.370220 6523 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:41.386946 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.407138 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.389811 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.391824 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.398473 5269 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 1.
I20260929 14:11:41.369393 6407 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:41.425385 5342 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 1510) took 1009 ms. Trace:
I20260929 14:11:41.426084 5342 rpcz_store.cc:276] 0929 14:11:40.415989 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:41.364715 (+948726us) service_pool.cc:227] Handling call
0929 14:11:41.377360 (+ 12645us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:41.401863 (+ 24503us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:41.417643 (+ 15780us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:41.425010 (+ 7367us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:41.425068 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:41.425151 (+ 83us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:41.425311 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260929 14:11:41.429734 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 116866us
I20260929 14:11:41.397944 5316 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:41.439316 6479 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:41.440071 6479 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:41.384279 6476 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:41.441560 6476 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:41.441969 6510 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:41.377832 6514 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:41.443054 6514 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:41.395339 5164 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:41.398373 5275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:50768 (request call id 863) took 1022 ms. Trace:
I20260929 14:11:41.391566 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.387080 5167 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:41.441330 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.464706 5276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50776 (request call id 1009) took 1106 ms. Trace:
I20260929 14:11:41.467089 5276 rpcz_store.cc:276] 0929 14:11:40.358203 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:40.390407 (+ 32204us) service_pool.cc:227] Handling call
0929 14:11:41.464491 (+1074084us) spinlock_profiling.cc:243] Waited 86.5 ms on lock 0x7b5400393d54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:41.464634 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":198885120}
I20260929 14:11:41.467691 6478 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:41.468003 5348 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 1504) took 1055 ms. Trace:
I20260929 14:11:41.468169 6478 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:41.468602 6478 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:41.448868 5275 rpcz_store.cc:276] 0929 14:11:40.376192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:40.814874 (+438682us) service_pool.cc:227] Handling call
0929 14:11:41.397629 (+582755us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b54003741d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2af0 00007fbda4a75f2a 00007fbda99b2c6e 00007fbda30a9807 00007fbda30a9792 00007fbda30a96c1 00007fbda30a963c 00007fbda30a88f2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:41.398061 (+ 432us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":684808576}
I20260929 14:11:41.395246 6521 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:41.469367 5349 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 1.
I20260929 14:11:41.416991 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:41.452075 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 56811.9us
I20260929 14:11:41.390931 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.379590 6495 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:41.468380 5348 rpcz_store.cc:276] 0929 14:11:40.412424 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:40.780088 (+367664us) service_pool.cc:227] Handling call
0929 14:11:40.823689 (+ 43601us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:40.975855 (+152166us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:40.978984 (+ 3129us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:40.982073 (+ 3089us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:40.982136 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:41.467660 (+485524us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b54002d3cd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:41.467711 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:41.467839 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":312687360}
I20260929 14:11:41.478468 5871 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:41.509975 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.510658 6515 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:41.485925 6389 raft_consensus.cc:2781] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:41.511094 6515 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:41.511402 6389 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:41.511500 6515 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:41.511919 6389 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:41.512928 6332 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:41.513597 6332 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:41.514066 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.486415 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:11:41.487804 6531 log.cc:924] Time spent T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.489s user 0.005s sys 0.000s
I20260929 14:11:41.488842 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.490173 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.314204 5192 rpcz_store.cc:276] 0929 14:11:39.823454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:40.810190 (+986736us) service_pool.cc:227] Handling call
0929 14:11:41.084008 (+273818us) spinlock_profiling.cc:243] Waited 1.93 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0dbce 000000000034aea9 00007fbda9939a30 00007fbda99ff7b4 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:11:41.084221 (+ 213us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:41.084494 (+ 273us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:41.084530 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:41.084577 (+ 47us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:41.084653 (+ 76us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:41.084709 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:41.084819 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4432128}
I20260929 14:11:41.492765 6416 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
W20260929 14:11:41.492765 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 66012.7us
I20260929 14:11:41.493911 6407 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:41.494956 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.425421 6523 raft_consensus.cc:3037] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:41.508831 5164 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:41.508987 6403 raft_consensus.cc:2726] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:41.516884 5314 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:41.549335 6416 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:41.551270 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.552238 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.554170 5199 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
W20260929 14:11:41.584892 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 76172.3us
W20260929 14:11:41.595991 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 79259.4us
W20260929 14:11:41.593729 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 206713us
W20260929 14:11:41.603063 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 205216us
I20260929 14:11:41.603730 5164 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:41.585497 6315 raft_consensus.cc:2781] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:41.611481 6315 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:41.612010 6315 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:41.589744 6386 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:41.610457 5316 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:41.618808 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.605472 6359 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:41.614202 5314 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:41.590291 5196 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:41.620185 6386 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:41.626029 6484 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:41.633729 6359 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:41.654194 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 47441.3us
I20260929 14:11:41.118558 5273 rpcz_store.cc:276] 0929 14:11:38.110398 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:38.134927 (+ 24529us) service_pool.cc:227] Handling call
0929 14:11:38.303523 (+168596us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:38.463737 (+160214us) raft_consensus.cc:1520] Early marking committed up to index 2
0929 14:11:40.798685 (+2334948us) log.cc:842] Serialized 14 byte log entry
0929 14:11:40.919646 (+120961us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:40.931677 (+ 12031us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:40.935722 (+ 4045us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:40.944038 (+ 8316us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:40.967411 (+ 23373us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1184345,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":161,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":139,"thread_start_us":2239895,"threads_started":2,"wal-append.queue_time_us":119326}
W20260929 14:11:41.661278 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 51540.9us
I20260929 14:11:41.635020 6427 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.647368 6386 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:41.669764 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.667815 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.671936 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.671005 5316 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:41.667815 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.649801 6403 raft_consensus.cc:2781] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:41.693094 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.694159 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.697001 5276 raft_consensus.cc:2454] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:41.699551 5342 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 1.
I20260929 14:11:41.703424 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.698935 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:41.709468 5271 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:41.711650 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.666447 6496 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:41.710525 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.720728 5271 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:41.670423 6391 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:41.673511 6510 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.735028 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.672112 6489 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260929 14:11:41.726776 5084 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 43563.2us
I20260929 14:11:41.730715 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.739473 5275 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:41.740638 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.695554 6403 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:41.732209 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.746572 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:41.749022 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 86874.3us
I20260929 14:11:41.745755 6332 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.713797 6411 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:41.760124 6411 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:41.760780 6411 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:11:41.765528 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 288177us
I20260929 14:11:41.763706 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.674919 6484 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:41.747753 6495 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.751488 5316 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:41.791595 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.711828 5351 raft_consensus.cc:2932] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:41.748220 6391 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:41.738029 6407 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.765959 6403 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:41.747965 6479 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.842438 6391 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:41.843150 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:41.860987 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 109939us
W20260929 14:11:41.865717 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 243199us
I20260929 14:11:41.869146 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.861699 5316 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:41.875360 6476 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:41.855136 6359 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:41.859848 6391 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:41.881989 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:41.892482 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 288861us
I20260929 14:11:41.886073 6493 raft_consensus.cc:2726] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:41.889786 5242 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:41.910068 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.869459 5871 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:41.911600 5270 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
W20260929 14:11:41.911765 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 53921.8us
I20260929 14:11:41.913055 6544 raft_consensus.cc:2781] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:41.913431 6332 raft_consensus.cc:2781] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:41.913511 6544 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:41.914063 6332 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:41.914175 6544 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:41.914638 6332 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:41.923854 6544 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:41.930773 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 69120us
I20260929 14:11:41.899142 6488 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:41.935079 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.891592 5273 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:41.946077 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:41.947983 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:41.945834 5164 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:41.938416 6472 consensus_meta.cc:287] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.576s user 0.019s sys 0.034s
I20260929 14:11:41.937845 6488 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:41.956748 6488 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:41.956806 6484 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:41.939420 6496 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:41.967217 6332 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:41.971426 6332 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:41.973757 5267 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:41.961994 6472 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.001837 6544 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:42.002647 6472 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:42.004936 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.019660 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:42.005055 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.951396 6523 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.040468 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:42.039973 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 107782us
I20260929 14:11:42.045818 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:41.973115 6391 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.061146 5164 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:41.993826 6496 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.063613 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.067294 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.067672 6446 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:42.068076 6391 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:42.068270 6446 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.070541 6446 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:42.071852 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.072644 5270 raft_consensus.cc:2379] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:42.064971 5195 raft_consensus.cc:1226] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
W20260929 14:11:42.043787 6469 consensus_meta.cc:287] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.678s user 0.013s sys 0.010s
I20260929 14:11:42.069010 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.079645 5198 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:42.084144 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.045008 6521 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:42.088595 6523 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:42.059056 6476 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:42.090466 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.090859 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:42.094642 5164 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:42.091660 6391 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:42.126369 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.090154 6521 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:42.130609 6521 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.135651 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.081530 6469 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.131824 6559 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:42.063670 5899 raft_consensus.cc:2932] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:42.139418 5351 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47258 (request call id 1240) took 1046 ms. Trace:
I20260929 14:11:42.163784 5351 rpcz_store.cc:276] 0929 14:11:41.083653 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:41.658410 (+574757us) service_pool.cc:227] Handling call
0929 14:11:41.675432 (+ 17022us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:41.705094 (+ 29662us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:11:41.878497 (+173403us) log.cc:842] Serialized 14 byte log entry
0929 14:11:42.125451 (+246954us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:42.125519 (+ 68us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:11:42.130158 (+ 4639us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:42.130243 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:42.130419 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":71385,"raft.queue_time_us":74043,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":16719,"spinlock_wait_cycles":2773760,"thread_start_us":154265,"threads_started":1,"wal-append.queue_time_us":252120}
I20260929 14:11:42.134354 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.142210 5183 tablet_service.cc:1514] Processing DeleteTablet for tablet 520249ed5be343dcb30c86d3396491a4 with delete_type TABLET_DATA_TOMBSTONED (TS ac17d3b553b74c9da39e0ab307addacd not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52180
I20260929 14:11:42.143625 5314 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:42.145897 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.153066 6559 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:42.130260 6391 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:42.166175 5346 raft_consensus.cc:1226] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:42.141140 5099 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52736 (request call id 220) took 6016 ms. Trace:
I20260929 14:11:42.170667 5276 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.143806 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.171550 6496 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:42.137379 6515 consensus_meta.cc:287] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.625s user 0.012s sys 0.038s
I20260929 14:11:42.172808 6559 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:42.175668 5314 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.159063 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:11:42.173743 6315 consensus_meta.cc:287] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.561s user 0.008s sys 0.020s
I20260929 14:11:42.174552 6391 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:42.164065 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.181434 5314 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:42.196779 5314 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.179095 6493 raft_consensus.cc:2781] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260929 14:11:42.201872 6389 consensus_meta.cc:287] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.689s user 0.008s sys 0.031s
I20260929 14:11:42.173921 6488 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.200709 6557 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.202296 6389 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.192309 5274 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.202010 6562 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.201542 6493 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:42.206746 6560 raft_consensus.cc:2781] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:42.184667 5193 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:42.209671 5316 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.209543 5314 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:42.178810 6515 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.213963 5165 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:42.185160 6315 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.233253 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.233767 6515 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:42.228071 6514 consensus_meta.cc:287] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.785s user 0.009s sys 0.012s
I20260929 14:11:42.210129 6493 raft_consensus.cc:3037] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.218480 5314 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.219014 6563 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:42.237382 6563 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:42.237571 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:42.237928 6563 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.238189 5900 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.238847 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.239202 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.239423 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.219895 5316 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.219705 6521 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.221724 6557 raft_consensus.cc:2781] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:42.240288 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.222067 5276 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
W20260929 14:11:42.223233 6478 consensus_meta.cc:287] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.754s user 0.014s sys 0.026s
I20260929 14:11:42.223554 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.241972 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.242067 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.210505 6488 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:42.243096 5314 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.243197 6476 raft_consensus.cc:2726] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260929 14:11:42.226006 6416 consensus_meta.cc:287] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.642s user 0.008s sys 0.013s
W20260929 14:11:42.226792 6561 log.cc:924] Time spent T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.093s user 0.003s sys 0.000s
W20260929 14:11:42.244386 6386 consensus_meta.cc:287] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.542s user 0.005s sys 0.008s
I20260929 14:11:42.208874 6560 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:42.244012 6416 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.244815 6560 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.244997 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.173934 5099 rpcz_store.cc:276] 0929 14:11:36.124332 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:36.124471 (+ 139us) service_pool.cc:227] Handling call
0929 14:11:42.140965 (+6016494us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:11:40.976012 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:11:41.067955 (+ 91943us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:11:41.068007 (+ 52us) write_op.cc:435] Acquired schema lock
0929 14:11:41.068020 (+ 13us) tablet.cc:660] Decoding operations
0929 14:11:41.068440 (+ 420us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:11:41.068535 (+ 95us) write_op.cc:641] Partition lock acquired for write op
0929 14:11:41.068578 (+ 43us) tablet.cc:683] Acquiring locks for 1 operations
0929 14:11:41.069865 (+ 1287us) tablet.cc:699] Row locks acquired
0929 14:11:41.069904 (+ 39us) write_op.cc:260] PREPARE: finished
0929 14:11:41.070124 (+ 220us) write_op.cc:270] Start()
0929 14:11:41.070342 (+ 218us) write_op.cc:276] Timestamp: P: 1790691101070019 usec, L: 0
0929 14:11:41.070380 (+ 38us) op_driver.cc:331] REPLICATION: starting
0929 14:11:41.082806 (+ 12426us) log.cc:842] Serialized 533 byte log entry
0929 14:11:41.893234 (+810428us) op_driver.cc:444] REPLICATION: finished
0929 14:11:42.002648 (+109414us) write_op.cc:301] APPLY: starting
0929 14:11:42.002745 (+ 97us) tablet.cc:1365] starting BulkCheckPresence
0929 14:11:42.002868 (+ 123us) tablet.cc:1368] finished BulkCheckPresence
0929 14:11:42.002991 (+ 123us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:11:42.003300 (+ 309us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:11:42.003545 (+ 245us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0929 14:11:42.003614 (+ 69us) write_op.cc:312] APPLY: finished
0929 14:11:42.003931 (+ 317us) log.cc:842] Serialized 23 byte log entry
0929 14:11:42.004117 (+ 186us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:42.004200 (+ 83us) write_op.cc:454] Released schema lock
0929 14:11:42.004269 (+ 69us) write_op.cc:341] FINISH: Updating metrics
0929 14:11:42.032710 (+ 28441us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:42.038005 (+ 5295us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":486245,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":109096,"apply.run_cpu_time_us":2412,"apply.run_wall_time_us":39422,"num_ops":1,"prepare.queue_time_us":187571,"prepare.run_cpu_time_us":36628,"prepare.run_wall_time_us":372013,"raft.queue_time_us":279928,"raft.run_cpu_time_us":7739,"raft.run_wall_time_us":13135,"replication_time_us":822691,"thread_start_us":436964,"threads_started":4,"wal-append.queue_time_us":473659}]]}
I20260929 14:11:42.236095 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.236151 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.236405 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.240160 5345 raft_consensus.cc:2379] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 2: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:42.241000 5165 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.242096 6478 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.244243 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.245198 5197 raft_consensus.cc:1226] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.235846 6540 raft_consensus.cc:2781] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:42.223695 6411 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.234230 6389 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:42.248867 5316 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.249503 5346 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47258 (request call id 1222) took 1268 ms. Trace:
I20260929 14:11:42.252211 5346 rpcz_store.cc:276] 0929 14:11:40.984315 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:41.410689 (+426374us) service_pool.cc:227] Handling call
0929 14:11:41.463050 (+ 52361us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:42.173471 (+710421us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:42.249294 (+ 75823us) spinlock_profiling.cc:243] Waited 703 ms on lock 0x7b54002526d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:42.249451 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1616356096}
I20260929 14:11:42.252857 5197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1208) took 1030 ms. Trace:
I20260929 14:11:42.253582 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.254655 6542 raft_consensus.cc:2726] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.249295 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.253242 5197 rpcz_store.cc:276] 0929 14:11:41.223075 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:41.947403 (+724328us) service_pool.cc:227] Handling call
0929 14:11:41.959505 (+ 12102us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:42.251135 (+291630us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:42.252653 (+ 1518us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b54003da754. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:42.252794 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":656608640}
I20260929 14:11:42.250697 5164 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.244755 6386 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.255077 5200 raft_consensus.cc:1226] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.266849 5201 raft_consensus.cc:1226] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.267297 6411 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:42.267560 6469 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:42.267956 6478 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:42.268608 5200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1207) took 1086 ms. Trace:
I20260929 14:11:42.268957 5200 rpcz_store.cc:276] 0929 14:11:41.194898 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:41.947641 (+752743us) service_pool.cc:227] Handling call
0929 14:11:41.966050 (+ 18409us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:42.266750 (+300700us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:42.268402 (+ 1652us) spinlock_profiling.cc:243] Waited 289 ms on lock 0x7b54001450d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:42.268554 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":664324352}
I20260929 14:11:42.269702 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.269312 6386 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:42.246021 6557 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:42.271342 6557 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:42.253918 5096 catalog_manager.cc:5495] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 2 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:42.272778 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.208885 6403 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.273824 5314 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:42.275266 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:42.254981 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.278280 6476 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.251936 6488 raft_consensus.cc:2726] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.285454 5870 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.282138 5239 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.249635 6540 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:42.287264 5164 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.288609 6554 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.290055 5165 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.291122 6403 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:42.291260 6416 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:42.291887 6479 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.292520 5239 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:42.302064 5239 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:42.302202 6508 raft_consensus.cc:2781] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:42.302693 6508 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:42.303165 6564 tablet_replica.cc:330] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd: stopping tablet replica
I20260929 14:11:42.292909 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:42.303655 5239 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.295562 6559 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.247138 6514 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.305042 5870 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.305471 6495 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:42.305866 6495 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:42.306222 6495 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.296981 6521 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:42.303130 6508 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
W20260929 14:11:42.307762 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 58987.4us
I20260929 14:11:42.307938 5096 catalog_manager.cc:5495] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 2 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:42.292490 6315 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:42.310619 5316 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.306676 6514 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:42.309511 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.311228 6359 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.298055 6540 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.313825 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.313825 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.330490 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:42.331789 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 58044.8us
I20260929 14:11:42.333663 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.332427 5314 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:42.307760 6515 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.337697 5870 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.329938 6416 raft_consensus.cc:2781] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:42.338992 6416 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:42.339111 5870 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.339435 6416 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.340126 6332 raft_consensus.cc:2726] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.340670 6332 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.335223 6550 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.313433 6555 raft_consensus.cc:2726] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.335937 5165 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.344650 5314 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:42.343901 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.346312 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.349779 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.351280 5351 raft_consensus.cc:1226] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.349711 6496 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.353704 5345 raft_consensus.cc:1226] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:42.352566 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.355759 5346 raft_consensus.cc:2379] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:42.358150 5350 raft_consensus.cc:1226] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.335284 6472 raft_consensus.cc:2781] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:42.361265 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:42.362250 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 51696.3us
I20260929 14:11:42.366202 5316 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.367353 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:42.369182 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.345619 6315 raft_consensus.cc:2781] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:42.367307 6403 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.374996 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.375548 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.376778 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.376878 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.349630 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.330960 6560 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.379935 5316 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.353796 6557 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.385022 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:42.386299 6557 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:42.380839 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.353451 6543 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:42.391132 6391 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.391846 5900 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:42.343327 6508 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.368219 6472 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:42.395787 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.394692 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.393952 5167 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:42.398815 5349 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:42.400938 6472 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.375332 6411 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:42.377914 6559 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:42.398607 5347 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
W20260929 14:11:42.403643 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 53251.2us
I20260929 14:11:42.387612 6488 raft_consensus.cc:2726] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.353627 6493 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.415730 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.416088 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:42.415041 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.417691 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.403625 6411 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:42.418156 6508 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:42.393196 6391 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:42.418471 6411 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.419046 6493 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:42.395556 6563 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.415936 5342 raft_consensus.cc:1226] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.376201 6315 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:42.424292 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:42.414757 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.425093 5314 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:42.424346 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.420894 5347 raft_consensus.cc:1226] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.426682 6563 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:11:42.389299 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 50264.1us
I20260929 14:11:42.427520 5349 raft_consensus.cc:2379] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:42.427587 5316 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.435971 5898 raft_consensus.cc:1226] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:42.436753 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.436832 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.424954 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:42.438092 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.438376 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.437515 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.441154 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.441774 5272 raft_consensus.cc:3037] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.442302 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.442889 5267 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:42.443109 6495 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.447330 6478 consensus_queue.cc:1046] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:11:42.447569 5870 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.448472 6566 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:42.448593 6478 raft_consensus.cc:3032] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:42.408231 6521 raft_consensus.cc:2781] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:42.453326 6478 raft_consensus.cc:729] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:42.453851 6521 raft_consensus.cc:686] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:42.454255 6478 consensus_queue.cc:260] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.412477 6514 raft_consensus.cc:2726] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.454803 6521 consensus_queue.cc:237] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.455601 6478 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:42.457368 6493 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:42.457790 6493 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:42.458175 6493 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.450691 5316 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.459573 5272 raft_consensus.cc:2454] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:42.461081 5165 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.463440 6332 raft_consensus.cc:2726] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.465049 5347 raft_consensus.cc:1226] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.424451 6315 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.468405 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:42.469326 5267 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:42.469705 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:42.471160 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.475291 5316 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:42.451071 6488 raft_consensus.cc:2781] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:42.479122 6564 raft_consensus.cc:2229] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:42.482246 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:42.482653 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:42.486374 5165 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.491492 5164 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.493865 5343 raft_consensus.cc:1226] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:42.477355 6559 raft_consensus.cc:2726] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.496286 5900 raft_consensus.cc:1226] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.499697 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:42.500665 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.501116 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.503394 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:42.503324 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.506655 6542 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.507350 6542 consensus_queue.cc:1046] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:42.507428 5164 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.508049 6564 raft_consensus.cc:2258] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:42.508602 5314 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:42.509351 6550 consensus_queue.cc:1046] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:42.509754 6554 raft_consensus.cc:3032] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:42.509961 6560 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:42.510187 6554 raft_consensus.cc:729] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:42.510466 6550 raft_consensus.cc:2726] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:42.511021 6554 consensus_queue.cc:260] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.511305 6493 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.509377 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:42.512349 6554 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.508733 5900 raft_consensus.cc:1226] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.514858 5316 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.516151 6493 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:42.517143 6559 raft_consensus.cc:2726] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.476917 6551 raft_consensus.cc:2726] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.485352 6403 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:42.518537 6403 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:42.518690 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.518950 6403 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:42.520592 5165 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.521380 6559 raft_consensus.cc:2726] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.522454 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.523782 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.524224 6564 ts_tablet_manager.cc:1905] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260929 14:11:42.525755 5898 raft_consensus.cc:1226] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.526444 6559 consensus_queue.cc:1046] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:42.526780 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.528141 6559 raft_consensus.cc:3032] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:42.528474 6559 raft_consensus.cc:729] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:42.527315 6546 raft_consensus.cc:2781] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:42.529745 6546 raft_consensus.cc:686] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:42.529590 6559 consensus_queue.cc:260] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.514777 5314 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:42.531314 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.532097 6555 raft_consensus.cc:2781] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:42.532305 6568 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:42.532608 6555 raft_consensus.cc:686] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:42.532826 6568 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.533521 6555 consensus_queue.cc:237] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.530814 6546 consensus_queue.cc:237] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.445160 6495 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:42.530745 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.539777 6568 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:42.539574 6389 consensus_queue.cc:1046] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:11:42.448947 5870 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.546804 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.547145 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.547478 5267 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.547998 6389 raft_consensus.cc:3032] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:42.548367 6389 raft_consensus.cc:729] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:42.549125 6389 consensus_queue.cc:260] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.514112 6488 raft_consensus.cc:686] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:42.553628 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.553946 6332 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260929 14:11:42.553437 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 104577us
I20260929 14:11:42.554664 5200 raft_consensus.cc:2379] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 2: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:42.554414 6488 consensus_queue.cc:237] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.555078 5870 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.557063 6332 raft_consensus.cc:2726] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.511091 6550 raft_consensus.cc:3032] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:42.558406 5870 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.558892 6550 raft_consensus.cc:729] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:42.560091 6550 consensus_queue.cc:260] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.561385 6567 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:42.561674 6386 consensus_queue.cc:1046] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:11:42.562036 6391 consensus_queue.cc:1046] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:42.563145 5164 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.563102 6391 consensus_queue.cc:1046] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:42.531327 6559 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.561901 6567 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.564291 6523 raft_consensus.cc:3032] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:42.564356 5164 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:42.567119 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 108597us
I20260929 14:11:42.570546 6542 raft_consensus.cc:2726] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:42.545279 6560 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:42.586575 5267 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:42.589249 5314 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:42.590158 6557 consensus_queue.cc:1046] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:11:42.591225 6542 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260929 14:11:42.591543 5873 leader_election.cc:334] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.558s (SENT)
I20260929 14:11:42.592199 5164 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.592953 6542 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:42.574842 6332 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.574097 6523 raft_consensus.cc:729] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:42.601562 6469 raft_consensus.cc:2781] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:42.593318 5870 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.601253 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.614256 5316 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.601620 5239 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:42.576604 6407 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:42.602450 6570 consensus_queue.cc:1046] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:42.603654 5274 raft_consensus.cc:2454] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:42.603724 6557 raft_consensus.cc:3032] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:42.600318 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.609042 6552 consensus_queue.cc:1046] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260929 14:11:42.610174 6523 consensus_queue.cc:260] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.616003 5316 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:42.571090 6386 consensus_queue.cc:1046] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260929 14:11:42.628104 6407 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.629108 6386 raft_consensus.cc:3032] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:42.629478 6386 raft_consensus.cc:729] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:42.623246 5239 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:42.595690 6411 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.630707 6552 raft_consensus.cc:3032] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:42.631107 6552 raft_consensus.cc:729] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:42.631928 6552 consensus_queue.cc:260] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.620805 6557 raft_consensus.cc:729] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:42.625788 6476 raft_consensus.cc:2726] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.613763 6550 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:42.630357 6386 consensus_queue.cc:260] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.634418 6523 raft_consensus.cc:3037] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.624727 6389 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.634938 5347 raft_consensus.cc:1226] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.630345 6407 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:42.632229 6540 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.619976 6332 raft_consensus.cc:2726] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.622136 6469 raft_consensus.cc:686] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:42.658599 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.631949 6411 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:42.662501 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.634876 6568 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:42.612954 5870 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.667083 6403 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.668283 6478 raft_consensus.cc:3032] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:42.668668 6478 raft_consensus.cc:729] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:42.669449 6403 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:42.670034 6495 raft_consensus.cc:3032] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:42.661500 6479 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:42.670415 6495 raft_consensus.cc:729] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:42.670388 5239 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.671566 6495 consensus_queue.cc:260] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.673203 6540 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:42.673018 6495 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:42.670738 6479 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:42.675877 5348 raft_consensus.cc:1226] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.676213 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:42.676399 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:42.676940 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.678797 5350 raft_consensus.cc:1226] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.617708 6416 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.677176 6508 raft_consensus.cc:2726] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:42.680284 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:42.649322 6555 raft_consensus.cc:2726] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260929 14:11:42.682291 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 66385.9us
I20260929 14:11:42.675889 6479 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.686079 5316 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.664235 6568 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:11:42.689013 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 77046.9us
I20260929 14:11:42.691803 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.692502 5316 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.695082 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.693878 5314 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:42.701910 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.702139 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.684978 6540 consensus_queue.cc:1046] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260929 14:11:42.704510 6563 raft_consensus.cc:2781] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:42.705025 6563 raft_consensus.cc:686] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:42.705799 6567 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:42.705932 6563 consensus_queue.cc:237] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.706640 6416 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:42.669492 6478 consensus_queue.cc:260] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.707847 6478 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.696210 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.710563 5316 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.710950 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.685734 6484 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.717231 5267 raft_consensus.cc:2379] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 2: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:42.687947 6386 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:42.692760 6568 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:42.704118 6540 raft_consensus.cc:3032] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:42.719831 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.688158 6555 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.721568 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.661249 6557 consensus_queue.cc:260] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.727286 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:42.716002 6562 raft_consensus.cc:2726] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:42.731339 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.734498 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:42.695785 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 80102.4us
I20260929 14:11:42.740180 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:42.737366 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.738288 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.743106 5343 raft_consensus.cc:2379] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 3: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:42.743469 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.742940 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.746657 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.756393 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.756734 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.728771 6540 raft_consensus.cc:729] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:42.760416 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.761613 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.761787 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.757809 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:42.762553 6552 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.764616 6472 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.696846 6521 consensus_queue.cc:1046] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260929 14:11:42.738822 6476 raft_consensus.cc:2726] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.776546 5345 raft_consensus.cc:1226] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.776757 5198 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:11:42.777372 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.759060 6359 consensus_queue.cc:1046] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
W20260929 14:11:42.782418 6572 log.cc:924] Time spent T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20260929 14:11:42.782657 6359 raft_consensus.cc:3032] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:42.783063 6359 raft_consensus.cc:729] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:42.784135 6359 consensus_queue.cc:260] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.788004 5165 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.789891 5350 raft_consensus.cc:1226] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.792873 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.793607 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.794106 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.794756 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.795159 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.796650 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.796764 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.797674 5196 raft_consensus.cc:2379] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 2: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:42.797860 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.798492 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.778076 6521 consensus_queue.cc:1225] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer d886fb1eb0474d2d9f7410c4013c45e4 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260929 14:11:42.804996 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.802827 5201 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:11:42.803175 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.806746 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.803869 5165 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.804281 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.804986 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.813987 5194 raft_consensus.cc:1226] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:42.813864 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.814540 5195 raft_consensus.cc:2454] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 1.
I20260929 14:11:42.815017 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.815330 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.815919 5195 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:42.816618 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.817770 5198 raft_consensus.cc:595] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.818303 5198 raft_consensus.cc:603] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:11:42.815677 5194 raft_consensus.cc:2362] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3. Re-sending same reply.
I20260929 14:11:42.755873 6469 consensus_queue.cc:237] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.821663 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.811061 5193 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:42.821391 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.825989 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.826620 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.828498 5873 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.811198 6359 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.830432 6315 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.830987 5193 raft_consensus.cc:1226] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:42.834941 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.833096 5197 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:42.809772 6550 raft_consensus.cc:2726] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.816442 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.835022 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.838650 5192 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:42.840898 6391 consensus_queue.cc:1046] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260929 14:11:42.844328 6391 raft_consensus.cc:3032] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:42.844378 6315 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:42.844772 6391 raft_consensus.cc:729] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:42.845086 5342 raft_consensus.cc:1226] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.820890 6521 raft_consensus.cc:2726] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.848742 5873 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.849383 6472 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:42.850443 5269 raft_consensus.cc:1226] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.854656 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.855777 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.856405 5272 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.820423 6540 consensus_queue.cc:260] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.857295 5270 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:42.857421 6332 raft_consensus.cc:2726] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.857704 6540 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.858532 5274 raft_consensus.cc:1226] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.860508 5274 raft_consensus.cc:1226] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.865192 6484 raft_consensus.cc:2726] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.845661 6391 consensus_queue.cc:260] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.874879 5272 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:42.876317 5873 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.877379 5314 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:42.884264 5873 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.891651 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.886945 6515 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.868847 6391 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.897886 6484 raft_consensus.cc:2781] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:42.898311 5342 raft_consensus.cc:1226] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.898423 6484 raft_consensus.cc:686] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:42.899356 6484 consensus_queue.cc:237] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.900143 6557 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.910215 6578 consensus_queue.cc:1046] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:42.910853 5192 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "675b77a0a88a4bf08134281fc0cd949e"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:11:42.910218 5344 raft_consensus.cc:1226] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.882602 6514 raft_consensus.cc:2781] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:42.911346 6578 raft_consensus.cc:3032] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:42.914077 6578 raft_consensus.cc:729] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:42.911082 6515 consensus_queue.cc:1046] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:11:42.915053 6566 raft_consensus.cc:3032] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:42.915572 6566 raft_consensus.cc:729] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:42.915005 6578 consensus_queue.cc:260] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.916471 6578 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.916388 6566 consensus_queue.cc:260] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:42.917675 6566 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:42.917761 6542 raft_consensus.cc:3032] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:42.911481 5192 raft_consensus.cc:595] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Received request to transfer leadership to TS 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:42.909384 5873 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.919785 5342 raft_consensus.cc:1226] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 2. Current term is 3. Ops: []
W20260929 14:11:42.920269 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 209778us
I20260929 14:11:42.878226 6479 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.922493 5316 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.921202 5873 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.923645 5316 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.926255 5198 raft_consensus.cc:1226] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.924382 6493 raft_consensus.cc:2726] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.918198 6542 raft_consensus.cc:729] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:42.931464 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.929919 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.927006 6332 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:42.922608 6515 raft_consensus.cc:2781] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:42.933916 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:42.920953 5192 raft_consensus.cc:603] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:11:42.935056 5273 raft_consensus.cc:1226] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.935113 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.921310 6514 raft_consensus.cc:686] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:42.913195 6386 consensus_queue.cc:1046] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260929 14:11:42.939955 5269 raft_consensus.cc:1226] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.935258 6315 raft_consensus.cc:2781] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:42.942058 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:42.967679 5193 raft_consensus.cc:1226] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.968499 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:42.971131 5196 raft_consensus.cc:2379] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 3: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:42.972036 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:42.973779 6386 raft_consensus.cc:3032] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:42.974164 6386 raft_consensus.cc:729] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:42.975008 6386 consensus_queue.cc:260] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.976207 6386 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
W20260929 14:11:42.861253 6389 consensus_queue.cc:1159] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260929 14:11:42.979759 6514 consensus_queue.cc:237] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.983546 6315 raft_consensus.cc:686] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:42.984527 6315 consensus_queue.cc:237] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.946520 6550 consensus_queue.cc:1046] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260929 14:11:42.986682 6550 consensus_queue.cc:1225] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer d886fb1eb0474d2d9f7410c4013c45e4 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260929 14:11:42.987142 6550 raft_consensus.cc:3032] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:42.987452 6550 raft_consensus.cc:729] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:42.942039 6479 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:42.943710 6568 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.945261 6542 consensus_queue.cc:260] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.998551 6542 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:42.988283 6550 consensus_queue.cc:260] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:42.948254 5275 raft_consensus.cc:1226] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:42.947504 6515 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:42.951002 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.954236 5165 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:42.957075 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:42.960052 5316 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:42.962891 6553 consensus_queue.cc:1046] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260929 14:11:42.994057 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 68681.3us
I20260929 14:11:43.001976 5271 raft_consensus.cc:1261] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260929 14:11:43.004571 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 50435.3us
I20260929 14:11:43.007107 5165 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.001555 6515 raft_consensus.cc:3037] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.009586 5165 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:43.013754 5165 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.013593 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:43.018673 5165 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.019393 6521 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.019929 6521 raft_consensus.cc:2726] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.021080 5165 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:43.022068 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 62099.4us
I20260929 14:11:43.022300 6389 raft_consensus.cc:2726] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.022083 5165 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:43.023487 6575 raft_consensus.cc:2726] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.023584 5316 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:43.023994 5165 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.025516 6577 raft_consensus.cc:2726] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.026435 6478 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:43.026332 6577 consensus_queue.cc:1046] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:43.026980 6478 raft_consensus.cc:2726] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.027366 6577 raft_consensus.cc:3032] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:43.027710 6577 raft_consensus.cc:729] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:43.028502 6577 consensus_queue.cc:260] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.031117 6577 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.035704 5165 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.036952 5201 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:43.039206 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:43.033499 6469 consensus_queue.cc:1035] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:43.005877 6523 raft_consensus.cc:2726] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.048635 5096 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44250 (request call id 122) took 6303 ms. Trace:
I20260929 14:11:43.049081 5096 rpcz_store.cc:276] 0929 14:11:36.745007 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:36.745330 (+ 323us) service_pool.cc:227] Handling call
0929 14:11:43.048577 (+6303247us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:11:42.497389 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:11:42.500930 (+ 3541us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:11:42.500968 (+ 38us) write_op.cc:435] Acquired schema lock
0929 14:11:42.500981 (+ 13us) tablet.cc:660] Decoding operations
0929 14:11:42.501389 (+ 408us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:11:42.501441 (+ 52us) write_op.cc:641] Partition lock acquired for write op
0929 14:11:42.501475 (+ 34us) tablet.cc:683] Acquiring locks for 2 operations
0929 14:11:42.501809 (+ 334us) tablet.cc:699] Row locks acquired
0929 14:11:42.501826 (+ 17us) write_op.cc:260] PREPARE: finished
0929 14:11:42.501960 (+ 134us) write_op.cc:270] Start()
0929 14:11:42.502066 (+ 106us) write_op.cc:276] Timestamp: P: 1790691102501922 usec, L: 0
0929 14:11:42.502082 (+ 16us) op_driver.cc:331] REPLICATION: starting
0929 14:11:42.502739 (+ 657us) log.cc:842] Serialized 913 byte log entry
0929 14:11:42.703588 (+200849us) op_driver.cc:444] REPLICATION: finished
0929 14:11:42.839157 (+135569us) write_op.cc:301] APPLY: starting
0929 14:11:42.839261 (+ 104us) tablet.cc:1365] starting BulkCheckPresence
0929 14:11:42.883312 (+ 44051us) tablet.cc:1368] finished BulkCheckPresence
0929 14:11:42.909798 (+ 26486us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:11:43.006954 (+ 97156us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:11:43.007130 (+ 176us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0929 14:11:43.007184 (+ 54us) write_op.cc:312] APPLY: finished
0929 14:11:43.007403 (+ 219us) log.cc:842] Serialized 29 byte log entry
0929 14:11:43.007576 (+ 173us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:43.007664 (+ 88us) write_op.cc:454] Released schema lock
0929 14:11:43.007704 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0929 14:11:43.008046 (+ 342us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:43.008083 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":126621,"apply.run_cpu_time_us":2869,"apply.run_wall_time_us":177871,"num_ops":2,"prepare.queue_time_us":109931,"prepare.run_cpu_time_us":2859,"prepare.run_wall_time_us":5619,"replication_time_us":191469,"thread_start_us":140643,"threads_started":2}]]}
I20260929 14:11:43.051529 6389 raft_consensus.cc:2726] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:43.056008 5165 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.056756 5898 raft_consensus.cc:1261] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:43.057356 6389 raft_consensus.cc:2726] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.058715 5165 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.059228 5097 catalog_manager.cc:5495] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 2 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:43.060909 6389 raft_consensus.cc:2726] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.062189 5165 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:43.066128 5195 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:43.067571 5165 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.069286 6523 consensus_queue.cc:1035] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:11:43.072031 5349 raft_consensus.cc:1226] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:43.072446 5344 raft_consensus.cc:1261] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:43.009668 6332 consensus_queue.cc:1046] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.060s
I20260929 14:11:43.073868 6495 consensus_queue.cc:1046] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:11:43.074326 6332 raft_consensus.cc:3032] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:43.075055 6508 raft_consensus.cc:3032] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:43.015277 6391 raft_consensus.cc:2726] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.077571 5165 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:43.075314 6332 raft_consensus.cc:729] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:43.075456 6508 raft_consensus.cc:729] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:43.017256 6540 raft_consensus.cc:2726] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:43.082412 5165 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.034042 6568 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:43.088163 6389 raft_consensus.cc:2726] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.088596 6521 raft_consensus.cc:2726] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:43.079478 6543 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.088750 6508 consensus_queue.cc:260] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:43.092140 6508 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:43.092182 6389 raft_consensus.cc:2726] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.101414 5271 raft_consensus.cc:1261] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260929 14:11:43.103099 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 86964.7us
I20260929 14:11:43.104794 5242 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:43.090387 6523 consensus_queue.cc:1035] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260929 14:11:43.120888 6332 consensus_queue.cc:260] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:43.114564 6551 consensus_queue.cc:1035] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
I20260929 14:11:43.113164 6495 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:43.122628 6495 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:43.123034 6495 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:43.129117 6583 raft_consensus.cc:3032] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:43.129592 6583 raft_consensus.cc:729] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:43.096712 5900 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:43.132766 6583 consensus_queue.cc:260] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.135968 5165 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.142998 6389 raft_consensus.cc:2781] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:43.143046 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:43.143702 6389 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:43.144230 6389 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:43.144726 5873 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:43.146803 5242 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.134158 6583 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.153803 5242 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.154372 6585 raft_consensus.cc:2726] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.156536 6508 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:43.157174 6568 raft_consensus.cc:2726] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.148909 6523 consensus_queue.cc:1046] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260929 14:11:43.158588 5242 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.159403 6508 raft_consensus.cc:2726] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.161680 5242 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.162683 5242 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.165647 5242 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.106314 6515 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:43.175984 5242 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.180243 5242 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.162360 5194 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "675b77a0a88a4bf08134281fc0cd949e"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:11:43.169701 6508 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:43.165930 6479 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.186914 5242 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:43.179189 6567 raft_consensus.cc:2726] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.172255 6446 raft_consensus.cc:2726] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.204702 5242 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.208910 5242 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.209730 6550 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.210969 6515 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:43.206693 6544 raft_consensus.cc:2726] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:43.175325 6564 ts_tablet_manager.cc:1918] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:43.214454 6578 raft_consensus.cc:2781] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:43.214711 6564 log.cc:1196] T 520249ed5be343dcb30c86d3396491a4 P ac17d3b553b74c9da39e0ab307addacd: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-0-root/wals/520249ed5be343dcb30c86d3396491a4
I20260929 14:11:43.214905 6578 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:43.215343 6578 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.216125 6564 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:52180 (request call id 738) took 1074 ms. Trace:
I20260929 14:11:43.218906 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:43.221069 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:43.221598 5195 raft_consensus.cc:3037] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.188748 6587 consensus_queue.cc:1046] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.029s
I20260929 14:11:43.228607 5199 raft_consensus.cc:1261] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:43.189419 6508 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:43.232643 5196 raft_consensus.cc:1261] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260929 14:11:43.232687 6587 raft_consensus.cc:3032] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:43.233273 6587 raft_consensus.cc:729] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:43.231174 5195 raft_consensus.cc:2454] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:43.233759 6332 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:43.235658 6587 consensus_queue.cc:260] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:43.237164 6587 raft_consensus.cc:3037] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.230324 5316 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:43.238844 5873 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:43.202056 6407 raft_consensus.cc:2726] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.223366 6568 raft_consensus.cc:2726] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.216495 6564 rpcz_store.cc:276] 0929 14:11:42.141700 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:42.141938 (+ 238us) service_pool.cc:227] Handling call
0929 14:11:42.284117 (+142179us) ts_tablet_manager.cc:1145] Deleting tablet 520249ed5be343dcb30c86d3396491a4
0929 14:11:42.302833 (+ 18716us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0929 14:11:42.831001 (+528168us) tablet_metadata.cc:641] Metadata flushed
0929 14:11:43.164561 (+333560us) tablet_metadata.cc:641] Metadata flushed
0929 14:11:43.216048 (+ 51487us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":81207,"tablet-delete.queue_time_us":92245,"thread_start_us":20285,"threads_started":1}
I20260929 14:11:43.234596 6508 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.250092 6555 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:43.242743 6515 raft_consensus.cc:2781] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:43.241359 5194 raft_consensus.cc:595] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Received request to transfer leadership to TS 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:43.253953 5194 raft_consensus.cc:603] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:11:43.253675 6568 consensus_queue.cc:1046] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:43.272100 6495 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:43.260660 6407 raft_consensus.cc:3032] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:43.273718 6407 raft_consensus.cc:729] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:43.273547 5242 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.274883 6567 raft_consensus.cc:2726] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.274641 6407 consensus_queue.cc:260] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.275980 6407 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.276814 5242 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.281574 5192 raft_consensus.cc:1261] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:43.281633 6446 raft_consensus.cc:2726] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.282986 5242 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.283931 6446 raft_consensus.cc:2726] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.257398 6515 raft_consensus.cc:686] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:43.285065 5242 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.285851 6446 raft_consensus.cc:2726] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.285594 6515 consensus_queue.cc:237] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:43.286995 5242 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.260334 6551 consensus_queue.cc:1035] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260929 14:11:43.267395 6568 consensus_queue.cc:1046] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.023s
I20260929 14:11:43.262013 6555 raft_consensus.cc:686] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:43.290736 6568 raft_consensus.cc:3032] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:43.234141 6514 consensus_queue.cc:1035] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260929 14:11:43.291275 6555 consensus_queue.cc:237] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:43.295019 6446 raft_consensus.cc:2781] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:43.295445 6446 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:43.295850 6446 raft_consensus.cc:3037] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:11:43.295526 6593 log.cc:924] Time spent T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.070s user 0.001s sys 0.000s
I20260929 14:11:43.303509 5086 catalog_manager.cc:4841] TS ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): tablet 520249ed5be343dcb30c86d3396491a4 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
W20260929 14:11:43.304147 5086 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 43575.6us
I20260929 14:11:43.308604 5274 raft_consensus.cc:1261] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:43.311084 5276 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:43.312597 5242 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.314750 5242 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:43.291229 6568 raft_consensus.cc:729] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:43.321568 5242 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.323076 6544 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:43.323683 6495 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:43.324800 6568 consensus_queue.cc:260] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:43.326436 6568 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.326273 6446 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.328300 6446 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:43.329731 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:43.331969 6557 consensus_queue.cc:1035] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:43.336899 5270 raft_consensus.cc:1261] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:43.339357 5276 raft_consensus.cc:1226] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:43.318109 6479 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:43.342100 5195 raft_consensus.cc:1261] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:43.349140 5342 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:11:43.350971 5242 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.349715 5342 raft_consensus.cc:595] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:43.343380 6479 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:43.346457 6446 raft_consensus.cc:2726] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.319386 6578 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.347141 6514 consensus_queue.cc:1035] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260929 14:11:43.352072 6493 consensus_queue.cc:1035] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260929 14:11:43.344895 6472 consensus_queue.cc:1035] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
I20260929 14:11:43.434422 5199 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260929 14:11:43.362231 6585 consensus_queue.cc:1046] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.076s
W20260929 14:11:43.445652 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 94780.6us
I20260929 14:11:43.447374 5199 raft_consensus.cc:1261] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:43.450706 5242 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.452258 5242 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.453378 5242 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.456823 5242 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.457804 6544 raft_consensus.cc:2726] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.458854 6578 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:43.459270 5276 raft_consensus.cc:1226] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:43.460093 6546 consensus_queue.cc:1035] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:43.460755 6567 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:43.461310 6567 raft_consensus.cc:686] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:43.364671 6479 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:43.462219 6567 consensus_queue.cc:237] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.463235 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:43.465663 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260929 14:11:43.469352 5273 raft_consensus.cc:1226] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:43.470256 6446 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.471441 5276 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.480527 5269 raft_consensus.cc:1226] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:43.475433 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:11:43.483994 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:43.484481 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:43.352394 6389 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.445959 6359 raft_consensus.cc:2726] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.488130 5197 raft_consensus.cc:1226] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:43.485276 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:43.486284 5165 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.486691 5276 raft_consensus.cc:1261] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:43.491142 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:43.488157 6575 consensus_queue.cc:1046] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260929 14:11:43.492837 6521 consensus_queue.cc:1046] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:43.493566 6540 raft_consensus.cc:2726] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.453209 6495 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.491194 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:43.498142 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:43.489773 5165 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.455662 6496 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260929 14:11:43.503913 6508 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.494666 6542 raft_consensus.cc:3032] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:43.519119 5165 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.517568 6542 raft_consensus.cc:729] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:43.523298 6550 consensus_queue.cc:1046] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:43.524296 6550 raft_consensus.cc:3032] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:43.523044 6542 consensus_queue.cc:260] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.527838 5242 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.528899 5242 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.530344 5271 raft_consensus.cc:1226] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:43.530251 5242 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.534102 6495 raft_consensus.cc:2726] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:43.526638 6386 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:43.535315 6386 raft_consensus.cc:686] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:43.536177 6386 consensus_queue.cc:237] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.537940 6566 raft_consensus.cc:3032] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:43.537983 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:43.538451 6566 raft_consensus.cc:729] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:43.539460 5899 raft_consensus.cc:1226] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:43.539253 6566 consensus_queue.cc:260] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:43.524614 6550 raft_consensus.cc:729] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:43.540508 6566 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:43.492357 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:11:43.527238 6479 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.541882 5097 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52750 (request call id 220) took 7488 ms. Trace:
I20260929 14:11:43.547758 5197 raft_consensus.cc:1261] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:43.540853 6550 consensus_queue.cc:260] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.575963 5242 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.577527 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:43.578439 5275 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260929 14:11:43.525565 6523 raft_consensus.cc:2726] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.580216 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:43.580502 5344 raft_consensus.cc:2379] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:43.542233 5097 rpcz_store.cc:276] 0929 14:11:36.057607 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:36.076230 (+ 18623us) service_pool.cc:227] Handling call
0929 14:11:43.541805 (+7465575us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:11:43.251354 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:11:43.296249 (+ 44895us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:11:43.296282 (+ 33us) write_op.cc:435] Acquired schema lock
0929 14:11:43.296295 (+ 13us) tablet.cc:660] Decoding operations
0929 14:11:43.296593 (+ 298us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:11:43.296650 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0929 14:11:43.296675 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0929 14:11:43.296844 (+ 169us) tablet.cc:699] Row locks acquired
0929 14:11:43.296857 (+ 13us) write_op.cc:260] PREPARE: finished
0929 14:11:43.296976 (+ 119us) write_op.cc:270] Start()
0929 14:11:43.297083 (+ 107us) write_op.cc:276] Timestamp: P: 1790691103296941 usec, L: 0
0929 14:11:43.297098 (+ 15us) op_driver.cc:331] REPLICATION: starting
0929 14:11:43.297744 (+ 646us) log.cc:842] Serialized 538 byte log entry
0929 14:11:43.431099 (+133355us) op_driver.cc:444] REPLICATION: finished
0929 14:11:43.431577 (+ 478us) write_op.cc:301] APPLY: starting
0929 14:11:43.431651 (+ 74us) tablet.cc:1365] starting BulkCheckPresence
0929 14:11:43.431741 (+ 90us) tablet.cc:1368] finished BulkCheckPresence
0929 14:11:43.431859 (+ 118us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:11:43.432141 (+ 282us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:11:43.432296 (+ 155us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0929 14:11:43.432343 (+ 47us) write_op.cc:312] APPLY: finished
0929 14:11:43.432509 (+ 166us) log.cc:842] Serialized 23 byte log entry
0929 14:11:43.432670 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:43.432746 (+ 76us) write_op.cc:454] Released schema lock
0929 14:11:43.432786 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0929 14:11:43.433021 (+ 235us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:43.433048 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":848,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":228,"apply.run_cpu_time_us":1669,"apply.run_wall_time_us":1668,"num_ops":1,"prepare.queue_time_us":159478,"prepare.run_cpu_time_us":2504,"prepare.run_wall_time_us":57508,"replication_time_us":133877,"thread_start_us":85528,"threads_started":1}]]}
I20260929 14:11:43.599881 6542 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.581678 6578 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.549147 5194 raft_consensus.cc:1261] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:43.530129 6587 raft_consensus.cc:2726] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:43.534322 6508 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:43.532814 6568 raft_consensus.cc:2726] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.550941 5099 catalog_manager.cc:5495] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260929 14:11:43.649838 6597 log.cc:924] Time spent T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.155s user 0.001s sys 0.000s
I20260929 14:11:43.650897 6550 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:43.580724 6493 consensus_queue.cc:1035] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.072s
W20260929 14:11:43.655215 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 55374.1us
I20260929 14:11:43.653681 6585 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:11:43.643983 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:43.644726 5242 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:43.549129 6389 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:43.658339 6472 raft_consensus.cc:979] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:11:43.698307 6472 raft_consensus.cc:1067] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:11:43.698217 6514 consensus_queue.cc:1035] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:43.581154 6567 consensus_queue.cc:1046] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.083s
I20260929 14:11:43.682009 5242 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.683271 5196 raft_consensus.cc:1226] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:43.684579 6407 raft_consensus.cc:2726] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:43.688546 6479 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:43.691088 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:43.704126 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:43.709664 5344 raft_consensus.cc:2379] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 3: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:43.702286 5197 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:11:43.710671 5197 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:43.711037 5197 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.709906 5195 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:43.713114 5242 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:43.705530 6544 raft_consensus.cc:2726] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:43.718235 6508 consensus_queue.cc:1046] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:11:43.722409 6563 raft_consensus.cc:3032] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:43.722777 6563 raft_consensus.cc:729] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:43.723541 6563 consensus_queue.cc:260] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.726408 5271 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:43.728534 5165 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:43.730247 6389 raft_consensus.cc:2781] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:43.730840 6389 raft_consensus.cc:686] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:43.731668 6389 consensus_queue.cc:237] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.725729 5197 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.734504 6508 consensus_queue.cc:1225] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer 675b77a0a88a4bf08134281fc0cd949e log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260929 14:11:43.734766 5197 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:43.736588 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:43.737164 5345 raft_consensus.cc:3032] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:43.738624 5197 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 14:11:43.737449 5345 raft_consensus.cc:729] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:43.743131 6555 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:11:43.739729 5345 consensus_queue.cc:260] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:43.748241 6611 log.cc:924] Time spent T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.053s user 0.000s sys 0.001s
I20260929 14:11:43.744652 5342 consensus_queue.cc:237] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:43.751024 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:43.756213 5271 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.751722 6563 raft_consensus.cc:3037] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:43.755064 5345 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.761354 5271 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:43.740343 6544 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:43.762816 5165 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:43.782590 5270 raft_consensus.cc:1261] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:43.769734 6542 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election won for term 2
W20260929 14:11:43.841398 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 56925.5us
I20260929 14:11:43.771404 6544 raft_consensus.cc:686] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:43.755818 6609 consensus_queue.cc:1046] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260929 14:11:43.879204 5898 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 14:11:43.880985 6407 raft_consensus.cc:3032] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:43.881393 6407 raft_consensus.cc:729] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:43.882265 6407 consensus_queue.cc:260] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:43.868835 6584 log.cc:924] Time spent T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260929 14:11:43.869431 6542 raft_consensus.cc:686] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:43.842549 6472 consensus_queue.cc:1035] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.045s
I20260929 14:11:43.892902 5274 raft_consensus.cc:1261] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:43.892655 6493 consensus_queue.cc:1046] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:43.894738 6544 consensus_queue.cc:237] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.895788 6552 raft_consensus.cc:3032] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:43.896139 6552 raft_consensus.cc:729] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:43.888687 6514 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
I20260929 14:11:43.897645 6407 raft_consensus.cc:3037] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.896963 6552 consensus_queue.cc:260] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.899963 6552 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:43.901444 6514 consensus_queue.cc:1035] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:43.894093 6542 consensus_queue.cc:237] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:43.933898 5096 catalog_manager.cc:5495] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 1 to 2, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 2 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:43.888561 5192 raft_consensus.cc:1226] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:43.948880 5200 raft_consensus.cc:1261] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:43.953480 5345 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:43.963280 5346 raft_consensus.cc:1261] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:43.888159 6403 consensus_queue.cc:1046] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
W20260929 14:11:44.027388 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 142235us
I20260929 14:11:44.030426 5197 raft_consensus.cc:1261] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:43.985412 6551 consensus_queue.cc:1035] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260929 14:11:44.039968 6624 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:43.985039 6479 consensus_queue.cc:1035] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
W20260929 14:11:44.088047 6615 log.cc:924] Time spent T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.060s user 0.001s sys 0.000s
I20260929 14:11:44.044296 6566 consensus_queue.cc:1035] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
W20260929 14:11:44.091526 5084 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet f84af62485a4405f887bb8675221641e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica c5ec116e39144881bdf1f2cd3e923cf5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:44.093330 5274 raft_consensus.cc:1261] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:44.085595 5192 raft_consensus.cc:1226] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:44.086583 6544 consensus_queue.cc:1035] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260929 14:11:44.124737 6587 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:44.125455 6587 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.127588 6587 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:44.128800 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:44.129441 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:44.130322 5900 raft_consensus.cc:2379] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 2: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:11:44.079932 6624 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.164155 5345 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:44.148547 6578 raft_consensus.cc:3032] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
W20260929 14:11:44.170303 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 57824.1us
W20260929 14:11:44.149529 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 168314us
I20260929 14:11:44.144981 6568 consensus_queue.cc:1046] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260929 14:11:44.175362 6568 raft_consensus.cc:3032] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
W20260929 14:11:44.181309 6621 log.cc:924] Time spent T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.147s user 0.001s sys 0.000s
W20260929 14:11:44.169916 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 87564.1us
I20260929 14:11:44.170027 6578 raft_consensus.cc:729] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:44.175868 5192 raft_consensus.cc:1226] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:44.175719 6568 raft_consensus.cc:729] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:44.351182 6550 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:44.282326 6521 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:44.385942 6476 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:44.388041 6642 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:44.378778 6536 log.cc:924] Time spent T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29: Append to log took a long time: real 0.096s user 0.000s sys 0.000s
I20260929 14:11:44.389691 6624 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:44.433943 6476 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.436069 6476 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:44.436335 6403 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:44.437063 6403 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.439531 6403 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:44.440495 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 250702us
I20260929 14:11:44.447655 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:44.448313 5275 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:44.447592 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:44.449805 5267 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:44.453442 5275 raft_consensus.cc:1226] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:44.456723 6389 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:44.457264 6389 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:44.360622 6568 consensus_queue.cc:260] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:44.359366 6578 consensus_queue.cc:260] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:44.433483 6583 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:44.392308 6550 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.461105 6496 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:44.396662 6521 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:44.416803 6642 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.532848 5239 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
W20260929 14:11:44.553580 6627 log.cc:924] Time spent T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.337s user 0.001s sys 0.000s
W20260929 14:11:44.554407 6637 log.cc:924] Time spent T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
W20260929 14:11:44.555735 6633 log.cc:924] Time spent T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.335s user 0.001s sys 0.001s
I20260929 14:11:44.496146 6568 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:44.556921 5314 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:44.533635 6563 consensus_meta.cc:287] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.771s user 0.005s sys 0.009s
I20260929 14:11:44.560576 5314 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:44.532976 6515 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:44.572221 6544 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:44.572867 6544 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:44.575531 6544 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:44.480566 6626 consensus_queue.cc:1046] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.084s
I20260929 14:11:44.483335 6583 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:44.582368 6521 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:44.530014 6496 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.566202 6508 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:44.586588 6508 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.568938 6562 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:44.590150 6562 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:44.590626 6562 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:44.552254 6624 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:44.593832 6624 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.602694 6642 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:44.603732 6638 log.cc:924] Time spent T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.120s user 0.000s sys 0.000s
I20260929 14:11:44.569666 6315 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:44.605641 6315 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:44.606051 6315 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:44.561882 6639 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:44.610139 6639 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.587212 5096 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52726 (request call id 224) took 1284 ms. Trace:
I20260929 14:11:44.608214 5097 catalog_manager.cc:5495] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 1 to 2, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 2 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:44.620095 5097 catalog_manager.cc:5495] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 2 to 3, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 3 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:11:44.622606 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:44.623365 6639 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:44.623508 5898 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:44.614234 5096 rpcz_store.cc:276] 0929 14:11:43.303280 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:43.303427 (+ 147us) service_pool.cc:227] Handling call
0929 14:11:44.587109 (+1283682us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:11:44.019980 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:11:44.092464 (+ 72484us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:11:44.092504 (+ 40us) write_op.cc:435] Acquired schema lock
0929 14:11:44.092524 (+ 20us) tablet.cc:660] Decoding operations
0929 14:11:44.092903 (+ 379us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:11:44.092961 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0929 14:11:44.092989 (+ 28us) tablet.cc:683] Acquiring locks for 1 operations
0929 14:11:44.093184 (+ 195us) tablet.cc:699] Row locks acquired
0929 14:11:44.093207 (+ 23us) write_op.cc:260] PREPARE: finished
0929 14:11:44.113877 (+ 20670us) write_op.cc:270] Start()
0929 14:11:44.133573 (+ 19696us) write_op.cc:276] Timestamp: P: 1790691104109763 usec, L: 0
0929 14:11:44.133608 (+ 35us) op_driver.cc:331] REPLICATION: starting
0929 14:11:44.174739 (+ 41131us) log.cc:842] Serialized 533 byte log entry
0929 14:11:44.218371 (+ 43632us) op_driver.cc:444] REPLICATION: finished
0929 14:11:44.219064 (+ 693us) write_op.cc:301] APPLY: starting
0929 14:11:44.219167 (+ 103us) tablet.cc:1365] starting BulkCheckPresence
0929 14:11:44.219303 (+ 136us) tablet.cc:1368] finished BulkCheckPresence
0929 14:11:44.219417 (+ 114us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:11:44.219637 (+ 220us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:11:44.219796 (+ 159us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0929 14:11:44.219842 (+ 46us) write_op.cc:312] APPLY: finished
0929 14:11:44.220028 (+ 186us) log.cc:842] Serialized 23 byte log entry
0929 14:11:44.220164 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:44.220238 (+ 74us) write_op.cc:454] Released schema lock
0929 14:11:44.220277 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0929 14:11:44.220513 (+ 236us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:44.220538 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":457,"apply.run_cpu_time_us":1693,"apply.run_wall_time_us":1692,"num_ops":1,"prepare.queue_time_us":20069,"prepare.run_cpu_time_us":3447,"prepare.run_wall_time_us":184700,"replication_time_us":77583}]]}
W20260929 14:11:44.631291 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 51560.8us
I20260929 14:11:44.597018 6624 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:44.633652 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:44.633708 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:44.634358 5351 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:44.634570 5345 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:44.636610 5239 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:44.563967 6646 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:44.641614 6587 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:44.642028 6587 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:44.642210 6555 raft_consensus.cc:3032] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:44.642858 6555 raft_consensus.cc:729] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:44.642377 6587 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:44.623927 6639 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:44.649240 5167 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:44.651110 6583 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:44.651607 6389 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:44.652388 6496 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:44.652827 6622 consensus_queue.cc:1046] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.006s
I20260929 14:11:44.654170 6550 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:44.611424 6515 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:44.648465 6552 consensus_meta.cc:287] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.748s user 0.009s sys 0.003s
I20260929 14:11:44.647825 6646 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:44.672920 6508 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:44.674043 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:44.674623 6515 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:44.674739 5898 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:44.675401 6646 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:44.670567 6642 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:44.676209 6642 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:44.676559 6642 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:44.678903 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:44.679010 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:44.679533 5271 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:44.679805 5270 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:44.684249 5167 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:44.685830 5351 raft_consensus.cc:1261] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:11:44.690783 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:44.691907 5345 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:44.689718 5314 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:44.703161 5342 raft_consensus.cc:1261] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:11:44.707813 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:44.708186 5342 raft_consensus.cc:1261] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:44.699342 6552 raft_consensus.cc:2781] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:44.669153 6639 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:44.715678 5316 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.511s user 0.010s sys 0.015s
W20260929 14:11:44.716022 5316 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.511s user 0.010s sys 0.015s
W20260929 14:11:44.716352 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 511895us
I20260929 14:11:44.720561 5271 raft_consensus.cc:1261] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:44.721200 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:44.721856 5349 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:44.721988 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:44.722564 5350 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:44.724406 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:44.721909 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:44.724983 5275 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:44.725301 5346 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:44.729374 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:44.735781 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:44.738055 5342 raft_consensus.cc:1261] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:44.705103 6650 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:44.743201 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:44.743402 5270 raft_consensus.cc:1261] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:11:44.716399 6552 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:44.744359 5164 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:44.743986 5345 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:44.702750 6508 consensus_queue.cc:1035] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260929 14:11:44.732306 6643 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:44.754289 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:44.710089 5870 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:44.740521 6542 consensus_queue.cc:1035] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260929 14:11:44.774245 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:44.774904 5901 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:44.761258 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:44.776494 5902 raft_consensus.cc:1261] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:44.777770 6472 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:44.778441 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:44.778510 6639 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:44.740123 6647 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:44.771008 6643 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.779374 6647 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.780191 6619 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:44.780658 6619 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:44.781294 6643 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:44.781399 6647 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:44.741330 6650 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:44.782440 5907 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:44.778264 6472 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:44.755229 5199 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
W20260929 14:11:44.790766 6631 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.553s user 0.010s sys 0.014s
I20260929 14:11:44.790701 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:44.791100 6640 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:44.791304 6631 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.605s user 0.010s sys 0.014s
I20260929 14:11:44.791832 6472 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:44.792060 6656 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:44.792577 6656 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.791572 6640 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:44.752468 6552 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:44.795463 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:44.800266 5271 raft_consensus.cc:2379] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:44.748225 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:44.745432 6386 consensus_queue.cc:1035] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.052s
I20260929 14:11:44.814066 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:44.815799 6521 raft_consensus.cc:2781] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:44.818816 6562 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.824338 6656 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:44.822630 6640 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:44.805997 6639 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:44.824335 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:44.843649 6639 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:44.844056 6639 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:44.827608 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:44.847078 6476 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:44.832712 6563 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:44.847543 6476 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:44.848218 6563 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:44.835250 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:44.849874 5271 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:44.850138 6563 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:44.788591 6650 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:44.843961 5275 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:11:44.825838 6587 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:44.852942 6587 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:44.854053 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:44.854764 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:44.855142 5345 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 3
I20260929 14:11:44.855324 5197 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:44.856084 6640 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:44.856667 6640 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:44.859082 6640 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:44.863613 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:44.864302 5273 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:44.866504 5314 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:44.867268 6546 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:44.867846 6551 raft_consensus.cc:2781] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:44.867724 6546 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:44.868230 6551 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 14:11:44.868281 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 121422us
I20260929 14:11:44.868579 6551 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:44.869750 6546 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:44.871094 6476 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:44.871870 6578 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:44.836447 6542 consensus_queue.cc:1035] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260929 14:11:44.874807 5165 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:44.875815 6389 consensus_queue.cc:1035] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:44.877591 6389 raft_consensus.cc:2781] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:44.878052 6389 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:44.878469 6389 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:44.886375 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:44.887084 5275 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:44.887002 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:44.887637 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:44.888091 5267 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:44.892782 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:44.893566 5271 raft_consensus.cc:2379] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 2: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:44.896400 5242 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:44.899387 6479 raft_consensus.cc:2781] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:44.899772 6479 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:44.900087 6479 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:44.902156 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:44.902804 5201 raft_consensus.cc:2454] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:44.902885 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:44.903865 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:44.904472 5197 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:44.904548 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:44.905427 5194 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:44.891711 6646 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.916985 6646 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:44.917557 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:44.921403 5314 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:44.922396 6635 raft_consensus.cc:2781] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:44.922762 6635 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:44.923084 6635 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:44.925243 6650 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.926229 5242 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:44.930286 6567 raft_consensus.cc:2781] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:44.930645 6567 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:44.931113 6567 raft_consensus.cc:3037] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:44.938234 5242 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:44.938252 5314 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:44.939239 5165 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:44.939806 6641 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:44.940212 6641 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:44.940515 6542 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:44.940528 6641 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:44.942435 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:44.943146 5905 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
W20260929 14:11:44.795897 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 92915.9us
I20260929 14:11:44.951390 5316 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:44.836540 6553 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:44.958407 5342 raft_consensus.cc:2354] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 2. Current term is 3.
I20260929 14:11:44.813968 5274 raft_consensus.cc:1261] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:44.967134 5345 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:44.969403 5239 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:44.955788 6553 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:44.839493 6521 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:44.979390 5197 raft_consensus.cc:1261] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260929 14:11:44.982941 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:44.984118 5276 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:44.967118 6493 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:44.966583 5193 raft_consensus.cc:1261] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:11:44.954216 6542 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:44.953982 6562 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:44.996135 5276 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:44.970017 6643 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:44.963423 5194 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:45.006299 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:45.018808 5197 raft_consensus.cc:2354] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4.
I20260929 14:11:45.007004 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:45.007674 6619 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:44.846138 6472 consensus_queue.cc:1035] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.146s
I20260929 14:11:44.935878 6479 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:44.884958 6551 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:44.980007 6656 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:44.980031 6446 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:44.993121 6521 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.009152 6567 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:45.021111 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 276804us
I20260929 14:11:44.953119 6315 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:45.016314 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:44.999074 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:44.996417 6332 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:45.062525 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:45.003646 6493 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:44.983013 6566 consensus_queue.cc:1035] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260929 14:11:45.063481 5199 raft_consensus.cc:2379] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 2: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:44.977452 6552 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.002215 6609 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:45.065501 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.022912 6643 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:45.019083 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:45.005647 6650 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:45.067210 5314 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:45.031828 5899 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:45.101421 5351 raft_consensus.cc:2354] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e for earlier term 3. Current term is 4.
I20260929 14:11:45.024466 6542 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.127359 5239 leader_election.cc:398] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 2. Current term is 3.
I20260929 14:11:45.099026 6643 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.131048 5239 leader_election.cc:401] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260929 14:11:45.075582 6493 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.140192 6642 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:45.141903 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:45.145675 5195 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:45.038619 6656 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.147462 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:45.101770 6609 raft_consensus.cc:686] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:45.150436 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:11:45.121735 6640 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:45.113857 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:45.154703 5195 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:45.157044 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:45.103858 6568 consensus_queue.cc:1035] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260929 14:11:45.060670 6446 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:45.093003 6661 consensus_queue.cc:1035] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.087s
I20260929 14:11:45.171892 6661 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:45.172465 6661 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.172962 5200 raft_consensus.cc:2379] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 3: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:45.091207 6332 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:45.094905 5871 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:45.177107 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:45.151197 6646 raft_consensus.cc:2726] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 2. Current term is 3.
I20260929 14:11:45.179566 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:45.181370 5198 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:45.187141 5199 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:45.195814 5272 raft_consensus.cc:1226] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:45.167328 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:45.201853 6446 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:45.174095 6640 raft_consensus.cc:686] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
W20260929 14:11:45.202028 6660 log.cc:924] Time spent T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.130s user 0.000s sys 0.000s
I20260929 14:11:45.209173 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:45.215006 5198 raft_consensus.cc:2454] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:45.175101 5194 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:45.213685 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:11:45.215724 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 152327us
I20260929 14:11:45.209721 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:45.225526 5272 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:45.218282 5199 raft_consensus.cc:3037] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.176988 6650 consensus_queue.cc:1035] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260929 14:11:45.199273 6652 raft_consensus.cc:2726] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:45.219497 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:45.247624 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.248677 5198 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.230214 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:45.250458 5349 raft_consensus.cc:1226] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260929 14:11:45.255630 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:45.257869 5084 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet f84af62485a4405f887bb8675221641e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica c5ec116e39144881bdf1f2cd3e923cf5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:45.257097 6568 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:45.263490 6667 log.cc:924] Time spent T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260929 14:11:45.264043 6661 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.264499 6619 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:45.263550 6568 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.264931 6619 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.266855 6619 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.267985 6661 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:11:45.227844 6672 log.cc:924] Time spent T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20260929 14:11:45.210469 6662 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:45.246811 6551 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:45.271271 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.194830 6389 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.202872 6493 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:45.217850 5193 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
W20260929 14:11:45.217846 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 148959us
I20260929 14:11:45.373049 6479 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:45.373903 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:45.351914 6542 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.347038 5192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1267) took 1061 ms. Trace:
I20260929 14:11:45.278270 6661 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:45.367031 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:45.218179 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:45.336851 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:45.251631 6553 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:45.266850 6568 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.269291 6567 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:45.345247 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:45.400174 5267 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.225561 6562 consensus_queue.cc:1046] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.110s
I20260929 14:11:45.328619 6315 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:45.329205 6552 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:45.216585 6446 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:45.349480 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:45.350689 6668 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:45.351974 6550 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:45.349521 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:45.417045 5344 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:45.346859 6619 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:45.347625 6536 log.cc:924] Time spent T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260929 14:11:45.218031 6609 consensus_queue.cc:237] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:45.364071 6666 log.cc:924] Time spent T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.130s user 0.000s sys 0.001s
I20260929 14:11:45.364778 6493 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:45.349433 6674 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:45.335985 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.345111 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.395195 5906 raft_consensus.cc:3037] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.386955 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:45.349874 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:45.403085 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:45.403591 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:45.405689 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:45.412422 5345 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.346284 6635 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.399590 6642 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:45.344683 6640 consensus_queue.cc:237] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:45.465420 5272 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:45.437978 5242 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:45.354874 6662 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.446928 5342 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:45.401685 5871 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:45.471757 5239 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:45.439898 6656 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.417533 6446 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.452490 6658 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:45.454032 6661 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:45.455052 6639 consensus_meta.cc:287] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.610s user 0.009s sys 0.000s
I20260929 14:11:45.455750 5274 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:45.416473 6668 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.448920 6674 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:45.459160 6641 consensus_meta.cc:287] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.511s user 0.009s sys 0.003s
I20260929 14:11:45.445425 6646 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:45.464471 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.433388 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:45.467906 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.447417 6332 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.473124 5314 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:45.484338 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 46444us
I20260929 14:11:45.429335 6550 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.437598 6619 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:45.496985 5267 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
W20260929 14:11:45.540824 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 58038.8us
I20260929 14:11:45.491734 5195 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:45.485692 6495 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:45.547842 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:45.469938 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.468483 5898 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.518713 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.553421 5345 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:45.576721 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:45.576954 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:45.594245 5197 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:11:45.582015 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 112277us
I20260929 14:11:45.597872 5200 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:45.547394 6661 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:45.490289 6618 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:45.486791 6479 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:45.549685 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:45.618259 6479 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:45.618449 5343 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.618793 6479 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:11:45.490957 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 95481.4us
I20260929 14:11:45.605748 5901 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:45.606369 6555 consensus_queue.cc:260] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.517609 6654 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:45.613476 6622 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:45.615026 6684 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:45.616343 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:45.640990 5345 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.617628 6446 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.623512 5314 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:45.623811 5199 raft_consensus.cc:2454] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:45.601699 5273 raft_consensus.cc:2379] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:45.645244 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:45.645530 5167 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:45.574656 6641 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.628335 5906 raft_consensus.cc:2454] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:45.387409 5192 rpcz_store.cc:276] 0929 14:11:44.283843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:44.284014 (+ 171us) service_pool.cc:227] Handling call
0929 14:11:44.360542 (+ 76528us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:44.565973 (+205431us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:44.568692 (+ 2719us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:44.635426 (+ 66734us) log.cc:842] Serialized 26 byte log entry
0929 14:11:44.941795 (+306369us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:44.941834 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:44.941915 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:45.335980 (+394065us) raft_consensus.cc:1671] finished
0929 14:11:45.336896 (+ 916us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:45.344985 (+ 8089us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":62313,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":170,"thread_start_us":374756,"threads_started":2,"wal-append.queue_time_us":356066}
I20260929 14:11:45.599432 6403 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:45.597165 6389 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.659323 5197 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:45.634110 6609 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:45.485543 6639 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:45.630267 5198 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:45.516809 6658 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.649495 5165 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:45.628033 6472 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:45.670728 6472 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.648129 6446 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:45.672009 6446 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.637555 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.663157 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:45.664073 6682 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:45.580587 6646 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:45.667794 5899 consensus_meta.cc:287] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.552s user 0.011s sys 0.007s
I20260929 14:11:45.627296 6661 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.647964 6622 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:45.668361 6550 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.668720 6662 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.670106 6668 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.677985 5350 raft_consensus.cc:2379] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 3: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:45.678454 5345 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:45.689096 5165 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:45.721942 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.716346 6671 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:45.637529 6654 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:45.652498 6315 raft_consensus.cc:2781] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:45.745920 5164 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:45.749562 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:45.752147 5272 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.633744 6683 consensus_queue.cc:1046] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.089s
I20260929 14:11:45.726060 5903 raft_consensus.cc:2379] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:11:45.692111 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:45.723577 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.755271 5276 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:45.628610 6618 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.673144 6472 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:45.662601 6684 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.717533 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.763814 5346 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:45.753976 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:45.674015 6446 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.766008 5164 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:45.767863 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:45.768625 5269 raft_consensus.cc:2379] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:45.769385 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:45.770117 5276 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.702497 6389 raft_consensus.cc:2726] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:45.717754 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:45.756333 5314 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:45.750816 6658 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:45.701833 6619 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:45.666671 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:45.702178 6403 raft_consensus.cc:686] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:45.776113 5267 raft_consensus.cc:2379] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 3: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:45.722249 6682 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:45.751034 6654 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.754187 6635 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:45.746136 6550 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:45.777105 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:45.793239 6641 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:11:45.794251 6680 log.cc:924] Time spent T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
I20260929 14:11:45.750550 6639 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.795650 6684 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.796977 6661 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.798552 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:45.799271 5199 raft_consensus.cc:2379] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 3: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
W20260929 14:11:45.800980 6643 consensus_meta.cc:287] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.649s user 0.012s sys 0.007s
I20260929 14:11:45.715435 5899 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:45.787470 5344 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:45.760089 6668 raft_consensus.cc:2781] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:45.839515 5316 leader_election.cc:398] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4.
I20260929 14:11:45.791579 6618 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.791642 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.792562 6550 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:45.802292 6622 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:45.802673 6583 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:45.850884 6583 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:45.851361 6583 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:45.805986 5871 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:45.806254 6332 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:11:45.807658 6521 consensus_meta.cc:287] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.744s user 0.014s sys 0.006s
I20260929 14:11:45.809865 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:45.811082 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:45.811854 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:45.815512 6646 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.787863 6682 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.816661 6403 consensus_queue.cc:237] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:45.719808 6557 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:45.677444 6609 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:45.830468 5272 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:45.854243 6550 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.875140 5314 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:45.724318 5348 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:45.790513 6674 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.778981 6683 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:45.844043 5316 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:45.845347 5164 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:45.846537 5907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45248 (request call id 209) took 1073 ms. Trace:
I20260929 14:11:45.847251 5242 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:45.848512 6668 raft_consensus.cc:686] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:45.848670 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.774106 6315 raft_consensus.cc:686] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:45.745834 6671 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.855475 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.859469 5906 raft_consensus.cc:2354] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4.
I20260929 14:11:45.846204 6472 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:45.873697 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:45.832674 5276 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:45.879601 5907 rpcz_store.cc:276] 0929 14:11:44.760789 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:44.760989 (+ 200us) service_pool.cc:227] Handling call
0929 14:11:45.833758 (+1072769us) spinlock_profiling.cc:243] Waited 6.73 ms on lock 0x7b540071ded4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:45.833891 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15482624}
I20260929 14:11:45.894006 5272 raft_consensus.cc:2354] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e for earlier term 2. Current term is 3.
I20260929 14:11:45.880352 5343 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:45.883788 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.895212 5344 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:45.885155 6544 raft_consensus.cc:2781] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:45.856639 6641 consensus_queue.cc:1035] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.024s
I20260929 14:11:45.885540 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.897114 5342 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:45.886668 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.888334 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:45.862239 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:45.868716 6639 raft_consensus.cc:2726] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260929 14:11:45.890829 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 45586.1us
I20260929 14:11:45.921528 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:45.922335 5201 raft_consensus.cc:2379] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 2: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:45.835074 6643 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.922803 5164 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:45.923715 6650 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:45.924172 6650 raft_consensus.cc:686] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:45.924327 6643 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.922781 6678 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:11:45.892437 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 48518us
W20260929 14:11:45.925796 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 64402.5us
I20260929 14:11:45.926147 5316 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:45.927174 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:45.927757 5198 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:45.928470 5316 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:45.924995 6678 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.900210 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:45.878360 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 72470.8us
I20260929 14:11:45.929592 5272 raft_consensus.cc:2379] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 2: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:45.929855 5871 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:45.930403 6678 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.931417 6629 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:45.931804 6629 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:45.932142 6629 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:45.917271 6407 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:45.866021 6332 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:45.928381 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:45.935527 5201 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.873379 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:45.936828 6639 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.924182 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:45.937849 5242 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:45.916702 6691 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:45.938376 5194 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.938906 5242 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
W20260929 14:11:45.937718 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 51066.7us
I20260929 14:11:45.867403 6521 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.902110 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:45.934096 6544 raft_consensus.cc:686] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:45.936579 6671 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:45.881896 6683 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:45.891543 6622 raft_consensus.cc:2781] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:45.876271 6557 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:45.944609 6692 log.cc:924] Time spent T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.078s user 0.001s sys 0.000s
I20260929 14:11:45.942716 6618 raft_consensus.cc:2781] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:45.943145 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:45.943917 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.944083 6567 raft_consensus.cc:2781] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:45.896541 6674 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:45.925052 6650 consensus_queue.cc:237] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:45.945760 5276 raft_consensus.cc:2379] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:45.896487 6635 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:45.948027 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:45.938656 6691 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:45.957736 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:45.934720 6407 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:45.958731 6332 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:45.998915 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 47967.9us
I20260929 14:11:45.958477 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:45.900753 6472 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:45.945428 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:45.951843 6683 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:45.991756 5316 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:45.992817 5201 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
W20260929 14:11:45.993368 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 46493.1us
I20260929 14:11:45.956059 6567 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:45.956663 6562 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:46.001609 5239 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:46.005895 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:46.013886 5275 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:45.956238 6668 consensus_queue.cc:237] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.009621 6685 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:45.675747 5870 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:46.010196 6557 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:45.998080 6622 raft_consensus.cc:686] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:46.010277 6544 consensus_queue.cc:237] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:46.011626 6407 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:46.013075 6567 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:45.994805 6674 raft_consensus.cc:686] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:46.027383 5239 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:46.037501 5871 leader_election.cc:398] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e for earlier term 2. Current term is 3.
I20260929 14:11:46.041747 5316 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:46.011902 6521 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.012729 6618 raft_consensus.cc:686] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:45.959744 6315 consensus_queue.cc:237] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.015198 6562 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:46.032212 6552 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260929 14:11:46.139168 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 111880us
W20260929 14:11:46.141146 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 92038.7us
I20260929 14:11:46.141203 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:46.143769 5906 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:46.039899 6619 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:46.146735 5242 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:46.039691 6678 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:11:46.160261 6479 consensus_meta.cc:287] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.525s user 0.007s sys 0.001s
I20260929 14:11:46.160403 5194 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:46.042930 5898 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:46.165872 5270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:50768 (request call id 884) took 1316 ms. Trace:
I20260929 14:11:46.166215 5270 rpcz_store.cc:276] 0929 14:11:44.837110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:44.837276 (+ 166us) service_pool.cc:227] Handling call
0929 14:11:46.152955 (+1315679us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b54003741d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2af0 00007fbda4a75f2a 00007fbda99b2c6e 00007fbda30a9807 00007fbda30a9792 00007fbda30a96c1 00007fbda30a963c 00007fbda30a88f2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:46.153441 (+ 486us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":285403520}
W20260929 14:11:46.055202 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 402122us
I20260929 14:11:46.170302 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:46.142725 6562 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:46.160730 6479 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.142421 6552 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:46.173333 6649 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:46.171123 5346 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:46.151141 6619 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:46.167837 6332 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:46.168126 6641 raft_consensus.cc:2726] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:11:46.187556 5269 raft_consensus.cc:2354] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3.
I20260929 14:11:46.054190 6685 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.195062 6685 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.196378 6583 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.201743 6472 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:46.202346 6609 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:46.203289 6697 log.cc:924] Time spent T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.179s user 0.001s sys 0.000s
I20260929 14:11:46.203431 5346 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:46.178921 6552 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:46.205175 5164 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:46.209376 5342 raft_consensus.cc:1226] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260929 14:11:46.210391 6521 raft_consensus.cc:2781] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:46.046522 6654 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:46.208000 6386 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:46.220323 6708 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:46.220750 6708 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:46.221055 6708 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:46.221335 6479 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.217151 5345 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:46.182956 6619 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:46.164626 6678 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:46.207440 6674 consensus_queue.cc:237] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:46.221482 6654 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.220353 6386 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.229717 6386 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.229928 6691 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:46.230376 6686 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:46.227375 6583 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:46.186169 6649 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 14:11:46.233835 5351 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38536 (request call id 65) took 1545 ms (client timeout 1750 ms). Trace:
W20260929 14:11:46.237038 5351 rpcz_store.cc:269] 0929 14:11:44.688089 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:44.689142 (+ 1053us) service_pool.cc:227] Handling call
0929 14:11:46.233660 (+1544518us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b54003f70d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:46.233783 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":889935360}
I20260929 14:11:46.217592 6641 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:46.245558 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:46.245647 6641 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:46.247514 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 205883us
I20260929 14:11:46.230918 6686 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.225183 5905 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:46.250890 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:46.252557 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:46.203992 6618 consensus_queue.cc:237] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:46.255762 5350 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1322) took 1240 ms (client timeout 1599 ms). Trace:
W20260929 14:11:46.256215 5350 rpcz_store.cc:269] 0929 14:11:45.015768 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:45.015991 (+ 223us) service_pool.cc:227] Handling call
0929 14:11:46.255578 (+1239587us) spinlock_profiling.cc:243] Waited 616 ms on lock 0x7b54002d1ed4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:46.255709 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1417340288}
I20260929 14:11:46.263221 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:46.269815 5344 raft_consensus.cc:2379] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 3: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:46.268061 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:46.257193 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:46.240800 6649 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:46.234894 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:46.284149 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:46.240561 6521 raft_consensus.cc:686] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:46.285271 5267 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:46.179553 6622 consensus_queue.cc:237] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.222013 6550 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.290525 5350 raft_consensus.cc:1261] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:11:46.295553 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:46.296190 5344 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:46.282635 6654 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:46.303855 5316 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:11:46.303267 6701 log.cc:924] Time spent T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
I20260929 14:11:46.318095 5314 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:46.323572 5344 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:46.324785 5239 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:46.329824 5346 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:11:46.330546 5346 raft_consensus.cc:595] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:46.314399 6678 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:46.317548 6641 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:46.330271 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:46.342274 6567 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.342893 6550 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.309965 6682 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:46.348233 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:46.348821 5351 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 3
I20260929 14:11:46.313342 6654 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:46.332870 6685 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:46.353384 5870 leader_election.cc:398] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e for earlier term 3. Current term is 4.
I20260929 14:11:46.339160 6472 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:46.324149 6609 consensus_queue.cc:1035] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260929 14:11:46.349035 6682 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:46.365046 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:11:46.350827 6706 log.cc:924] Time spent T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.103s user 0.001s sys 0.000s
W20260929 14:11:46.369227 6705 log.cc:924] Time spent T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20260929 14:11:46.373515 6711 raft_consensus.cc:2726] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:46.332280 6521 consensus_queue.cc:237] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.376610 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:46.377256 5270 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:46.380272 5351 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:46.381704 5164 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:46.379092 5314 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:46.372030 6472 raft_consensus.cc:686] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:46.369609 6685 raft_consensus.cc:686] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
W20260929 14:11:46.390447 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 57622us
I20260929 14:11:46.392343 5316 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:46.392830 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:46.397742 5904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45248 (request call id 216) took 1525 ms. Trace:
I20260929 14:11:46.398145 5904 rpcz_store.cc:276] 0929 14:11:44.871994 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:44.873940 (+ 1946us) service_pool.cc:227] Handling call
0929 14:11:46.359577 (+1485637us) spinlock_profiling.cc:243] Waited 79.6 ms on lock 0x7b5800027698. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:11:46.369561 (+ 9984us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:46.382971 (+ 13410us) raft_consensus.cc:1520] Early marking committed up to index 6
0929 14:11:46.393339 (+ 10368us) log.cc:842] Serialized 14 byte log entry
0929 14:11:46.397389 (+ 4050us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:46.397446 (+ 57us) raft_consensus.cc:1631] Marking committed up to 6
0929 14:11:46.397490 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:46.397559 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:46.397668 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":94,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"spinlock_wait_cycles":183041792,"thread_start_us":10599,"threads_started":2}
I20260929 14:11:46.400964 5270 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
mode: GRACEFUL
new_leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
from {username='slave'} at 127.0.0.1:48196
I20260929 14:11:46.382000 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:46.401638 5270 raft_consensus.cc:595] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Received request to transfer leadership to TS d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:46.384749 6682 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:46.384632 6625 raft_consensus.cc:2741] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260929 14:11:46.416853 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50770 (request call id 1557) took 1450 ms. Trace:
I20260929 14:11:46.414166 5316 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:46.417366 5271 rpcz_store.cc:276] 0929 14:11:44.961423 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:44.962365 (+ 942us) service_pool.cc:227] Handling call
0929 14:11:44.962739 (+ 374us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:44.963429 (+ 690us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:44.963470 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:46.011204 (+1047734us) spinlock_profiling.cc:243] Waited 4.76 ms on lock 0x7b2c0004c1b8. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda3450e11 00007fbda3460442 00007fbda4aa8c11 00007fbda4a6d58a 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772
0929 14:11:46.011512 (+ 308us) log.cc:842] Serialized 26 byte log entry
0929 14:11:46.031195 (+ 19683us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:46.031233 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:46.031297 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:46.402126 (+370829us) raft_consensus.cc:1671] finished
0929 14:11:46.402147 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:46.411830 (+ 9683us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":178,"raft.queue_time_us":322,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":12739,"spinlock_wait_cycles":10955008,"thread_start_us":18543,"threads_started":1,"wal-append.queue_time_us":177153}
I20260929 14:11:46.400307 6567 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.384913 6629 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.424451 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:46.425097 6715 log.cc:924] Time spent T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.056s user 0.001s sys 0.000s
I20260929 14:11:46.384750 6562 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.427037 5195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52188 (request call id 1284) took 1152 ms. Trace:
I20260929 14:11:46.428540 5195 rpcz_store.cc:276] 0929 14:11:45.274667 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:45.366726 (+ 92059us) service_pool.cc:227] Handling call
0929 14:11:46.426896 (+1060170us) spinlock_profiling.cc:243] Waited 97.8 ms on lock 0x7b540023b9d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:46.426984 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":225006464}
I20260929 14:11:46.434371 5346 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:11:46.415215 6508 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:46.434765 6718 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:46.434962 5346 raft_consensus.cc:595] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:46.434566 5902 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45248 (request call id 218) took 1536 ms. Trace:
I20260929 14:11:46.436036 5165 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:46.436955 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:46.419019 6682 raft_consensus.cc:686] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:46.446373 5344 raft_consensus.cc:1226] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:46.446575 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:46.391562 6649 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:46.418004 6686 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.435092 6508 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:46.458343 6508 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:46.437628 6718 raft_consensus.cc:686] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:46.457859 6649 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:46.441696 5316 leader_election.cc:398] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Vote denied by peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497) with higher term. Message: Invalid argument: T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4.
I20260929 14:11:46.440419 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:46.448020 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:46.471174 5270 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:46.448519 6719 raft_consensus.cc:2781] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:46.471927 5316 leader_election.cc:401] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260929 14:11:46.450245 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:46.453166 6657 raft_consensus.cc:2726] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:46.453330 6685 consensus_queue.cc:237] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.475770 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:46.472126 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:46.460666 6683 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.473927 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:46.477785 5271 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:46.476352 5906 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:46.477959 5165 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:46.476559 5276 raft_consensus.cc:2354] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:11:46.454336 6654 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.482105 5165 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:46.473958 6550 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:46.484061 6661 raft_consensus.cc:2726] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:46.484189 6550 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.482807 6668 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:46.486341 6550 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.487314 5165 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:46.488158 5165 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:46.489244 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 558243us
I20260929 14:11:46.461989 6718 consensus_queue.cc:237] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.468312 6508 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:46.498311 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:11:46.500816 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 59272.1us
I20260929 14:11:46.493275 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:46.502223 5316 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:46.504312 5165 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:46.510424 5346 raft_consensus.cc:2354] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3.
I20260929 14:11:46.495219 6683 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.523531 5165 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:46.523947 5906 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:11:46.524163 5164 leader_election.cc:398] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3.
I20260929 14:11:46.524919 5165 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:46.526603 5242 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:46.525607 5164 leader_election.cc:401] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260929 14:11:46.486814 5905 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:46.479429 6719 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:46.528141 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:46.530262 5165 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:46.445704 6472 consensus_queue.cc:237] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.536000 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:46.538887 5271 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
W20260929 14:11:46.483356 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 128833us
I20260929 14:11:46.505304 6657 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:46.504540 6550 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:46.510252 6403 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497) with higher term. Message: Invalid argument: T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4.
I20260929 14:11:46.548593 5165 leader_election.cc:398] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:11:46.557195 5165 leader_election.cc:401] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260929 14:11:46.525377 6671 raft_consensus.cc:2726] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:46.528682 6661 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:46.540707 6711 raft_consensus.cc:2726] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:46.541314 6587 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:46.536684 6521 raft_consensus.cc:2726] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:46.546339 6657 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:46.546264 6650 raft_consensus.cc:2726] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:46.541100 6719 raft_consensus.cc:3037] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:46.510160 6508 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.693535 6661 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:46.571614 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:46.524725 6708 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.734100 6716 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:46.698110 6657 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:46.734711 6716 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:46.699149 6587 raft_consensus.cc:686] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:46.438335 5902 rpcz_store.cc:276] 0929 14:11:44.897896 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:44.898046 (+ 150us) service_pool.cc:227] Handling call
0929 14:11:46.287393 (+1389347us) spinlock_profiling.cc:243] Waited 55.5 ms on lock 0x7b5800027698. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:11:46.302559 (+ 15166us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:46.377927 (+ 75368us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:46.382403 (+ 4476us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:46.384534 (+ 2131us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:11:46.386351 (+ 1817us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:46.427842 (+ 41491us) spinlock_profiling.cc:243] Waited 64.3 ms on lock 0x7b54005ad7d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:46.427893 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:46.434507 (+ 6614us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":275667072}
I20260929 14:11:46.752094 6508 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.788717 5270 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:46.690343 6552 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.690021 5870 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:46.794180 5348 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47252 (request call id 1597) took 1330 ms. Trace:
W20260929 14:11:46.792918 6639 consensus_meta.cc:287] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.848s user 0.012s sys 0.000s
I20260929 14:11:46.724243 6629 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:46.564380 6682 consensus_queue.cc:237] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.809659 6716 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.810506 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:46.811431 5270 raft_consensus.cc:2454] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 1.
I20260929 14:11:46.786636 6562 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:46.813860 5871 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:46.800786 5348 rpcz_store.cc:276] 0929 14:11:45.464110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:45.467661 (+ 3551us) service_pool.cc:227] Handling call
0929 14:11:46.790763 (+1323102us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b540012c854. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:46.794098 (+ 3335us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":279537920}
I20260929 14:11:46.816851 6587 consensus_queue.cc:237] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:46.818964 5903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45232 (request call id 252) took 1243 ms. Trace:
I20260929 14:11:46.821336 6728 raft_consensus.cc:2781] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:46.821790 6728 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:46.822175 6728 raft_consensus.cc:3037] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
W20260929 14:11:46.824561 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 256303us
I20260929 14:11:46.819432 5903 rpcz_store.cc:276] 0929 14:11:45.575643 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:45.576410 (+ 767us) service_pool.cc:227] Handling call
0929 14:11:46.818739 (+1242329us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b54000cbcd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:46.818896 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":273213184}
I20260929 14:11:46.816411 6716 raft_consensus.cc:2726] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:46.813861 6639 raft_consensus.cc:2726] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:46.852083 6675 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:46.888721 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:46.892416 5903 raft_consensus.cc:1261] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260929 14:11:46.895404 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:11:46.900580 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 388592us
I20260929 14:11:46.908501 6711 raft_consensus.cc:979] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:11:46.911686 6711 raft_consensus.cc:1067] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Signalling peer c5ec116e39144881bdf1f2cd3e923cf5 to start an election
W20260929 14:11:46.885435 6726 log.cc:924] Time spent T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.193s user 0.000s sys 0.001s
I20260929 14:11:46.880604 5273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1560) took 1476 ms. Trace:
I20260929 14:11:46.916486 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:46.879981 6671 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:46.883778 6722 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:46.922748 5270 raft_consensus.cc:1261] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260929 14:11:46.903825 6567 consensus_queue.cc:1035] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260929 14:11:46.927691 5275 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:50776
I20260929 14:11:46.928426 5275 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:46.928900 5275 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:46.932531 6722 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:46.916545 6650 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:46.916122 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:46.919075 6708 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.921588 6552 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:46.916354 6683 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:46.923856 6662 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:46.929436 6550 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:46.940016 5345 raft_consensus.cc:1261] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:11:46.924268 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:46.956677 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 412466us
I20260929 14:11:46.940438 6659 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:46.964241 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:46.929163 6671 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:46.957829 6661 raft_consensus.cc:2726] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3.
I20260929 14:11:46.972806 6722 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.962258 6662 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.985337 6684 consensus_queue.cc:1046] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:46.986523 6662 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.990907 6407 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:46.990473 6658 raft_consensus.cc:2932] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:46.991611 6407 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:46.994355 6407 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:46.995200 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:46.957190 6552 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:46.918941 5273 rpcz_store.cc:276] 0929 14:11:45.402705 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:45.402829 (+ 124us) service_pool.cc:227] Handling call
0929 14:11:46.874459 (+1471630us) spinlock_profiling.cc:243] Waited 87 ms on lock 0x7b5400392954. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:46.878609 (+ 4150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":200023424}
W20260929 14:11:47.009703 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 78588.9us
I20260929 14:11:47.010713 5164 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:46.961804 6675 raft_consensus.cc:2726] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:11:46.972404 5903 raft_consensus.cc:2932] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:46.946909 6650 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:46.984064 6619 consensus_meta.cc:287] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.748s user 0.013s sys 0.000s
W20260929 14:11:47.023039 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 474610us
I20260929 14:11:46.954188 6683 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:47.026362 6675 raft_consensus.cc:2781] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:47.026790 6675 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:47.027143 6675 raft_consensus.cc:3037] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:47.027500 6683 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:47.029702 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 128655us
I20260929 14:11:46.974597 5198 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:47.034251 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1197) took 1215 ms. Trace:
I20260929 14:11:47.034621 5199 rpcz_store.cc:276] 0929 14:11:45.818993 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:47.016978 (+1197985us) service_pool.cc:227] Handling call
0929 14:11:47.033798 (+ 16820us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:47.033956 (+ 158us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:47.033978 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:47.034015 (+ 37us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:47.034040 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:47.034100 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:47.034201 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:47.035244 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:47.037282 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:47.038259 5198 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:47.018162 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:11:47.044435 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 121328us
I20260929 14:11:47.046476 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:47.048005 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:47.048820 5198 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:47.049846 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:47.050582 5198 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:47.051638 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:47.147329 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 103291us
I20260929 14:11:47.147668 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:47.150168 5351 raft_consensus.cc:1261] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260929 14:11:47.152434 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:47.153908 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:47.155144 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:47.156428 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:47.157387 5197 raft_consensus.cc:2379] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 3: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:46.981745 6550 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:47.151158 6674 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:47.201376 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:46.981668 6659 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:47.032261 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:11:47.251703 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 205327us
W20260929 14:11:47.256276 5084 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet f84af62485a4405f887bb8675221641e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
W20260929 14:11:47.252436 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 204554us
I20260929 14:11:47.023066 6552 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:47.285806 5347 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:11:47.285805 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:47.287734 5197 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:11:47.288954 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:47.032161 6619 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:47.279402 5199 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:47.281490 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:47.179858 6668 consensus_meta.cc:287] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.694s user 0.007s sys 0.004s
I20260929 14:11:47.176959 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:47.336133 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:47.161326 6567 consensus_queue.cc:1046] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.142s
I20260929 14:11:47.257911 6654 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:47.324728 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:47.174117 6731 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:47.326423 5271 raft_consensus.cc:2354] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3.
I20260929 14:11:47.338860 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:47.336992 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:47.341966 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:47.342908 6678 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:47.331918 6655 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:47.345741 6668 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:47.280118 6583 consensus_queue.cc:1035] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260929 14:11:47.303295 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:47.354338 5348 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:47.324193 5902 raft_consensus.cc:1261] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260929 14:11:47.248096 6674 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:47.281031 6661 consensus_queue.cc:1035] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260929 14:11:47.387496 6446 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:47.388334 6552 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:47.388443 5901 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45252 (request call id 370) took 1045 ms. Trace:
I20260929 14:11:47.360394 6654 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:47.390326 6736 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:47.391412 5274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 84) took 1571 ms. Trace:
I20260929 14:11:47.532593 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:47.392436 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
W20260929 14:11:47.392880 6719 consensus_meta.cc:287] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.687s user 0.007s sys 0.003s
I20260929 14:11:47.371968 6678 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:47.360620 6731 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:47.397207 6728 consensus_meta.cc:287] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.575s user 0.008s sys 0.002s
I20260929 14:11:47.398413 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52182 (request call id 1198) took 1478 ms. Trace:
W20260929 14:11:47.407620 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 56762.7us
W20260929 14:11:47.426004 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 207378us
I20260929 14:11:47.562299 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:47.376151 5193 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:47.470247 5197 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:11:47.375082 6567 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:47.401556 6587 raft_consensus.cc:3032] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:47.526437 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:47.552508 6662 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:47.366901 6708 consensus_queue.cc:1035] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.113s
W20260929 14:11:47.590708 5905 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45248 (request call id 221) took 1781 ms (client timeout 2308 ms). Trace:
I20260929 14:11:47.407281 6671 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:47.591120 5905 rpcz_store.cc:269] 0929 14:11:45.809529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:45.810773 (+ 1244us) service_pool.cc:227] Handling call
0929 14:11:47.590539 (+1779766us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x7b5400631ad4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:47.590652 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":830127360}
I20260929 14:11:47.561758 5196 rpcz_store.cc:276] 0929 14:11:45.920181 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:45.933800 (+ 13619us) service_pool.cc:227] Handling call
0929 14:11:47.398185 (+1464385us) spinlock_profiling.cc:243] Waited 313 ms on lock 0x7b540031e554. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:47.398348 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":719303424}
I20260929 14:11:47.593024 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:47.593700 5900 raft_consensus.cc:1261] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260929 14:11:47.594105 5196 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:11:47.594769 5196 raft_consensus.cc:595] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:47.596469 5902 raft_consensus.cc:1261] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260929 14:11:47.598410 6678 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:47.599968 6731 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:47.522475 5901 rpcz_store.cc:276] 0929 14:11:46.343340 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:46.346013 (+ 2673us) service_pool.cc:227] Handling call
0929 14:11:47.387572 (+1041559us) spinlock_profiling.cc:243] Waited 4.74 ms on lock 0x7b5800027698. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:11:47.387822 (+ 250us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:47.388016 (+ 194us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:47.388050 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:47.388138 (+ 88us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:11:47.388169 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:47.388250 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:47.388383 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10906880}
I20260929 14:11:47.602229 5192 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:47.550449 5274 rpcz_store.cc:276] 0929 14:11:45.819576 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:45.920387 (+100811us) service_pool.cc:227] Handling call
0929 14:11:45.920667 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:46.012742 (+ 92075us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:46.015733 (+ 2991us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:46.028236 (+ 12503us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:11:46.031511 (+ 3275us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:47.391122 (+1359611us) spinlock_profiling.cc:243] Waited 80.2 ms on lock 0x7b54003b0e54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:47.391182 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:47.391343 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":184349568}
I20260929 14:11:47.604569 5097 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44250 (request call id 123) took 3868 ms. Trace:
I20260929 14:11:47.604765 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:47.607529 5904 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:11:47.608402 5274 raft_consensus.cc:1261] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260929 14:11:47.608765 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:47.580688 6587 raft_consensus.cc:729] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:47.610100 6587 consensus_queue.cc:260] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:47.611085 6587 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:47.611972 6619 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:47.551569 5195 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:11:47.555357 5316 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:47.479873 6722 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:47.576675 5164 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:47.620617 5351 raft_consensus.cc:1226] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260929 14:11:47.482542 6683 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:47.588058 5898 raft_consensus.cc:2354] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:11:47.631160 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:47.632395 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:47.633249 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 77981.4us
I20260929 14:11:47.482016 6649 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:47.579684 5200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52182 (request call id 1205) took 1045 ms. Trace:
I20260929 14:11:47.635553 5200 rpcz_store.cc:276] 0929 14:11:46.534540 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:46.537444 (+ 2904us) service_pool.cc:227] Handling call
0929 14:11:47.562147 (+1024703us) spinlock_profiling.cc:243] Waited 6.1 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:11:47.579617 (+ 17470us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14034176}
I20260929 14:11:47.578295 5199 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:47.380975 6655 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:47.583379 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:47.634431 5316 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:47.530326 5242 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
W20260929 14:11:47.584937 5275 consensus_meta.cc:287] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.653s user 0.007s sys 0.000s
I20260929 14:11:47.552628 6568 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:47.549944 6650 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:47.589597 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:47.650115 5270 raft_consensus.cc:2354] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4.
W20260929 14:11:47.650704 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 120455us
W20260929 14:11:47.628887 6657 consensus_meta.cc:287] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.841s user 0.007s sys 0.006s
I20260929 14:11:47.625337 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:47.626339 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:47.560322 6719 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:47.604935 5097 rpcz_store.cc:276] 0929 14:11:43.735710 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:43.735849 (+ 139us) service_pool.cc:227] Handling call
0929 14:11:47.604482 (+3868633us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:11:45.126987 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:11:45.266303 (+139316us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:11:45.266341 (+ 38us) write_op.cc:435] Acquired schema lock
0929 14:11:45.266353 (+ 12us) tablet.cc:660] Decoding operations
0929 14:11:45.266761 (+ 408us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:11:45.266819 (+ 58us) write_op.cc:641] Partition lock acquired for write op
0929 14:11:45.266852 (+ 33us) tablet.cc:683] Acquiring locks for 2 operations
0929 14:11:45.267174 (+ 322us) tablet.cc:699] Row locks acquired
0929 14:11:45.267188 (+ 14us) write_op.cc:260] PREPARE: finished
0929 14:11:45.267329 (+ 141us) write_op.cc:270] Start()
0929 14:11:45.267436 (+ 107us) write_op.cc:276] Timestamp: P: 1790691105267284 usec, L: 0
0929 14:11:45.267453 (+ 17us) op_driver.cc:331] REPLICATION: starting
0929 14:11:45.268089 (+ 636us) log.cc:842] Serialized 913 byte log entry
0929 14:11:45.675160 (+407071us) op_driver.cc:444] REPLICATION: finished
0929 14:11:45.916104 (+240944us) write_op.cc:301] APPLY: starting
0929 14:11:45.916195 (+ 91us) tablet.cc:1365] starting BulkCheckPresence
0929 14:11:45.916305 (+ 110us) tablet.cc:1368] finished BulkCheckPresence
0929 14:11:45.916444 (+ 139us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:11:45.916790 (+ 346us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:11:45.916949 (+ 159us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0929 14:11:45.916990 (+ 41us) write_op.cc:312] APPLY: finished
0929 14:11:45.917185 (+ 195us) log.cc:842] Serialized 29 byte log entry
0929 14:11:45.917372 (+ 187us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:45.917460 (+ 88us) write_op.cc:454] Released schema lock
0929 14:11:45.917499 (+ 39us) write_op.cc:341] FINISH: Updating metrics
0929 14:11:45.917729 (+ 230us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:45.917755 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":176752,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":240667,"apply.run_cpu_time_us":1914,"apply.run_wall_time_us":1916,"num_ops":2,"prepare.queue_time_us":387518,"prepare.run_cpu_time_us":3259,"prepare.run_wall_time_us":174323,"replication_time_us":407613,"thread_start_us":465829,"threads_started":2}]]}
I20260929 14:11:47.628684 5873 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:47.574548 6407 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:47.689747 5314 leader_election.cc:398] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4.
I20260929 14:11:47.690019 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:47.690236 5314 leader_election.cc:401] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260929 14:11:47.656737 5351 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
W20260929 14:11:47.692633 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 64067.3us
I20260929 14:11:47.693665 5873 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:47.641091 5275 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:47.588281 6550 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:47.588444 6747 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:47.651677 5242 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:47.623771 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:47.524816 6446 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:47.563241 6728 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:47.559844 6744 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.120s
I20260929 14:11:47.555189 6736 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:47.700194 5197 raft_consensus.cc:1261] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260929 14:11:47.721788 6407 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:47.629119 6661 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260929 14:11:47.755239 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 103690us
I20260929 14:11:47.656603 6643 raft_consensus.cc:2726] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:47.629602 6731 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:47.643115 6619 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:47.644985 6557 raft_consensus.cc:2726] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:47.801420 5316 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:47.801043 5275 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:47.660584 6657 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:47.771207 6756 raft_consensus.cc:2781] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260929 14:11:47.891448 6752 log.cc:924] Time spent T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.254s user 0.001s sys 0.000s
I20260929 14:11:47.696346 6711 raft_consensus.cc:2726] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:47.756861 6515 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260929 14:11:47.881879 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 43585.8us
I20260929 14:11:47.659422 5898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45232 (request call id 256) took 1305 ms. Trace:
I20260929 14:11:47.658185 6659 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:47.619401 6662 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:47.756488 6639 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:47.889578 5242 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:47.723129 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 68686.9us
W20260929 14:11:47.890151 5269 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1568) took 2110 ms (client timeout 2146 ms). Trace:
I20260929 14:11:47.639638 6734 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.184s
I20260929 14:11:47.756927 5194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1207) took 1185 ms. Trace:
I20260929 14:11:47.636847 6722 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:47.946861 6655 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:47.948293 6732 raft_consensus.cc:3032] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:47.948704 6732 raft_consensus.cc:729] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:47.649091 6683 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:47.949237 6755 log.cc:924] Time spent T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260929 14:11:47.948324 6722 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:47.950069 6763 raft_consensus.cc:2726] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:47.950762 6683 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:47.897876 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 262744us
I20260929 14:11:47.638293 6746 consensus_queue.cc:1046] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.251s
I20260929 14:11:47.648941 6649 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:47.881983 5165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 2.146s (SENT) [suppressed 40 similar messages]
I20260929 14:11:47.658689 5870 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:47.689541 6650 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.199s
I20260929 14:11:47.644609 6609 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:47.901189 6550 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260929 14:11:47.987569 6719 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:47.960136 6683 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:47.924137 5164 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:47.972445 6493 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:48.024322 5906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45248 (request call id 225) took 1493 ms. Trace:
I20260929 14:11:48.024837 5906 rpcz_store.cc:276] 0929 14:11:46.531452 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:46.531921 (+ 469us) service_pool.cc:227] Handling call
0929 14:11:46.532291 (+ 370us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:46.772366 (+240075us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:46.772410 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:46.772466 (+ 56us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:11:46.772509 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:48.024028 (+1251519us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b54005ae954. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:48.024083 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:48.024251 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":551566080}
I20260929 14:11:47.904381 6661 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:47.654067 6568 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:47.923228 6407 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:47.935101 5201 raft_consensus.cc:2354] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3.
W20260929 14:11:47.936241 5269 rpcz_store.cc:269] 0929 14:11:45.779697 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:45.793372 (+ 13675us) service_pool.cc:227] Handling call
0929 14:11:47.889911 (+2096539us) spinlock_profiling.cc:243] Waited 327 ms on lock 0x7b54003b0454. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:47.890083 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":752978176}
I20260929 14:11:47.905352 6753 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:47.943089 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 53583.1us
I20260929 14:11:48.070662 5269 raft_consensus.cc:1226] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:48.071564 5242 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:47.826594 6747 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:47.955191 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:47.921207 6315 raft_consensus.cc:2726] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260929 14:11:47.961344 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 73573.1us
I20260929 14:11:47.915102 6756 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:11:47.938189 6658 consensus_meta.cc:287] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.903s user 0.012s sys 0.006s
I20260929 14:11:47.964919 6446 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:47.935751 6639 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:11:47.948724 6761 log.cc:924] Time spent T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260929 14:11:47.901362 6731 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:47.917774 6758 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:48.074276 6407 raft_consensus.cc:2726] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:47.990150 6657 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:47.923091 6674 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:47.908252 6619 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:47.943833 6654 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:47.886449 5196 raft_consensus.cc:1261] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260929 14:11:47.916608 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 115284us
I20260929 14:11:48.034233 6683 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:47.588034 6567 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:47.946201 5194 rpcz_store.cc:276] 0929 14:11:46.535104 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:46.543043 (+ 7939us) service_pool.cc:227] Handling call
0929 14:11:46.994007 (+450964us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0dbce 000000000034aea9 00007fbda9939a30 00007fbda99ff7b4 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:11:46.994172 (+ 165us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:47.036443 (+ 42271us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:47.039430 (+ 2987us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:47.045396 (+ 5966us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:11:47.158663 (+113267us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:47.650887 (+492224us) spinlock_profiling.cc:243] Waited 34.3 ms on lock 0x7b5400352854. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:47.654784 (+ 3897us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:47.720263 (+ 65479us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":327341056}
I20260929 14:11:47.924903 5898 rpcz_store.cc:276] 0929 14:11:46.350310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:46.350869 (+ 559us) service_pool.cc:227] Handling call
0929 14:11:47.392286 (+1041417us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b5800027698. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0dbce 000000000034aea9 00007fbda9939a30 00007fbda99ff7b4 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:11:47.655413 (+263127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8452352}
I20260929 14:11:47.987666 6762 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:48.039853 6741 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:48.040334 6770 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:48.046758 5193 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:11:48.039566 6661 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:48.047466 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:48.116701 5273 raft_consensus.cc:3037] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
W20260929 14:11:48.116890 6675 consensus_meta.cc:287] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 1.089s user 0.012s sys 0.001s
I20260929 14:11:48.117583 6661 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:48.050568 5348 consensus_meta.cc:287] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.529s user 0.008s sys 0.000s
I20260929 14:11:48.054912 6583 raft_consensus.cc:979] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:11:48.059708 6728 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:48.065335 6739 log.cc:924] Time spent T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.750s user 0.004s sys 0.008s
W20260929 14:11:48.065898 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 183927us
W20260929 14:11:48.066385 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 154221us
I20260929 14:11:48.073331 5242 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:48.131983 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 58745.9us
I20260929 14:11:48.132917 5242 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:48.019194 6609 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:47.986509 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 348345us
I20260929 14:11:48.079797 6639 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:48.120956 6770 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.083756 6769 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:48.085336 5316 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:48.020249 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 96193.8us
I20260929 14:11:48.073616 6753 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:48.099082 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1299) took 1070 ms. Trace:
I20260929 14:11:48.038430 6493 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:48.099907 6619 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:48.100075 6403 consensus_queue.cc:1046] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:48.076372 6756 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:48.081007 6758 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.115032 5167 leader_election.cc:398] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Vote denied by peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497) with higher term. Message: Invalid argument: T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:11:48.116143 6747 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.117367 6675 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.117905 6683 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:48.101487 6642 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:48.119820 6741 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:48.085529 6736 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:48.122876 6649 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.126870 6567 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.135342 5346 raft_consensus.cc:1226] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:48.138170 5351 raft_consensus.cc:1261] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:11:48.136574 5194 raft_consensus.cc:1261] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:11:48.139276 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:48.122028 5348 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
W20260929 14:11:48.150513 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 65301.7us
I20260929 14:11:48.151350 5164 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:48.191690 5099 catalog_manager.cc:5495] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 3 to 4, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:11:48.152743 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:11:48.233098 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 125162us
I20260929 14:11:48.234381 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.235877 5099 catalog_manager.cc:5495] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 2 to 3, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 3 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:48.148768 5242 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:48.237381 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:11:48.235117 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 83828.3us
W20260929 14:11:48.237823 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 89122.6us
I20260929 14:11:48.153209 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.130942 6718 raft_consensus.cc:1067] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:11:48.239302 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.238456 6476 raft_consensus.cc:2726] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4.
I20260929 14:11:48.241705 6662 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:48.241809 6770 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.243026 6772 raft_consensus.cc:2726] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:48.243566 6753 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:48.232484 5196 rpcz_store.cc:276] 0929 14:11:47.028850 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:47.596114 (+567264us) service_pool.cc:227] Handling call
0929 14:11:47.691757 (+ 95643us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:48.098733 (+406976us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:48.099002 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:48.274014 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.274544 5274 raft_consensus.cc:1261] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:11:48.274686 5272 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:48.274849 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:48.273559 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:48.275615 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:48.276005 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.276616 5272 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f84af62485a4405f887bb8675221641e"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:48196
I20260929 14:11:48.277377 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:48.249147 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:48.279174 5196 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:11:48.249714 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:48.279819 5898 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:48.280848 5196 raft_consensus.cc:1261] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:11:48.250349 6550 raft_consensus.cc:2726] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:48.248970 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:48.254015 5242 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:48.237608 5099 catalog_manager.cc:5495] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 2 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:11:48.282982 5099 catalog_manager.cc:5495] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 2 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:11:48.290285 5242 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:48.292622 5272 raft_consensus.cc:595] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:48.288007 6762 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:48.283946 6732 consensus_queue.cc:260] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.293560 6753 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:48.284919 5099 catalog_manager.cc:5495] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 2 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:11:48.255476 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.137627 6508 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:48.256436 5350 raft_consensus.cc:1261] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260929 14:11:48.191629 6587 consensus_meta.cc:287] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.580s user 0.009s sys 0.002s
I20260929 14:11:48.260952 5198 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:48.255437 6657 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:48.235201 6741 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:48.154011 6583 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:48.265008 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.263957 5904 raft_consensus.cc:1261] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260929 14:11:48.184906 6658 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:47246 (request call id 540) took 4640 ms. Trace:
I20260929 14:11:48.253986 6619 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:48.270743 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:48.344388 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.306445 6555 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:48.187691 6609 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.190073 6568 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:48.313830 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:48.253948 5348 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1358) took 1222 ms. Trace:
I20260929 14:11:48.342638 6583 raft_consensus.cc:686] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:48.295989 5314 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:48.313035 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:48.308610 5242 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:48.353912 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:48.312222 5201 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52182 (request call id 1209) took 1776 ms (client timeout 2201 ms). Trace:
W20260929 14:11:48.365721 5276 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41626 (request call id 88) took 1874 ms (client timeout 2044 ms). Trace:
I20260929 14:11:48.306716 6550 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260929 14:11:48.308674 5200 raft_consensus.cc:2932] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
W20260929 14:11:48.368827 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 72898.4us
I20260929 14:11:48.298933 6446 raft_consensus.cc:2726] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:48.267736 6639 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:48.188587 6654 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:48.239984 6642 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:48.272375 6748 log.cc:924] Time spent T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.793s user 0.000s sys 0.011s
I20260929 14:11:48.350402 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.298250 5273 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:48.299916 5099 catalog_manager.cc:5495] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 2 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:11:48.297330 6479 consensus_queue.cc:1035] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260929 14:11:48.355937 6758 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.354611 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.390178 5351 raft_consensus.cc:2379] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:48.389197 5195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1307) took 1132 ms. Trace:
I20260929 14:11:48.357717 5899 raft_consensus.cc:2379] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 2: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:11:48.348927 6662 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:48.346704 6770 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:48.353529 5348 rpcz_store.cc:276] 0929 14:11:47.031142 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:47.033285 (+ 2143us) service_pool.cc:227] Handling call
0929 14:11:48.238463 (+1205178us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:48.309206 6643 consensus_queue.cc:1035] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260929 14:11:48.362519 6493 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:48.205904 6769 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:48.314342 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260929 14:11:48.365746 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 57255.8us
W20260929 14:11:48.365619 5201 rpcz_store.cc:269] 0929 14:11:46.535627 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:46.535791 (+ 164us) service_pool.cc:227] Handling call
0929 14:11:48.312012 (+1776221us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b5400350f54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:48.312159 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3204644608}
I20260929 14:11:48.272397 6708 raft_consensus.cc:979] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5: : Instructing follower d886fb1eb0474d2d9f7410c4013c45e4 to start an election
I20260929 14:11:48.293984 6762 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:48.366281 5276 rpcz_store.cc:269] 0929 14:11:46.491483 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:48.312743 (+1821260us) service_pool.cc:227] Handling call
0929 14:11:48.365665 (+ 52922us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:48.371057 5350 raft_consensus.cc:2379] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:48.376269 6751 raft_consensus.cc:2781] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:48.374104 6472 consensus_queue.cc:1035] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260929 14:11:48.376565 6583 consensus_queue.cc:237] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.379776 6639 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.380627 5903 raft_consensus.cc:2379] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 4: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:11:48.379787 6683 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:48.429656 6683 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:48.430289 6685 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:11:48.374280 6654 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:48.368285 6609 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:48.394356 5099 catalog_manager.cc:5495] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 2 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
W20260929 14:11:48.404296 5899 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45252 (request call id 374) took 1931 ms (client timeout 1916 ms). Trace:
I20260929 14:11:48.398751 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.438058 5099 catalog_manager.cc:5495] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 2 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:11:48.440292 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:48.441118 5346 raft_consensus.cc:2354] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:11:48.394496 5195 rpcz_store.cc:276] 0929 14:11:47.257342 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:47.627911 (+370569us) service_pool.cc:227] Handling call
0929 14:11:48.250612 (+622701us) spinlock_profiling.cc:243] Waited 1.99 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:11:48.250781 (+ 169us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:48.250956 (+ 175us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:11:48.268530 (+ 17574us) log.cc:842] Serialized 14 byte log entry
0929 14:11:48.388582 (+120052us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:48.388747 (+ 165us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:11:48.388878 (+ 131us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:48.388972 (+ 94us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:48.389127 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":17547,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":122,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"spinlock_wait_cycles":4577152,"thread_start_us":136879,"threads_started":2}
I20260929 14:11:48.401525 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:48.441855 5200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1308) took 1159 ms. Trace:
I20260929 14:11:48.403617 5314 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:48.442270 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:48.391937 6732 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:48.446483 5164 leader_election.cc:398] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:11:48.446794 5164 leader_election.cc:401] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260929 14:11:48.446244 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.410948 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:11:48.447882 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:11:48.390288 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:48.449513 5196 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:48.393324 6641 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:48.397923 6718 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:48.415747 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.396806 6650 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:48.417697 5871 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:48.459856 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.467765 6741 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.402470 6770 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.431926 6683 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:48.436172 6654 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.412845 6662 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.416652 6568 consensus_queue.cc:1035] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260929 14:11:48.442199 5200 rpcz_store.cc:276] 0929 14:11:47.282659 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:47.636367 (+353708us) service_pool.cc:227] Handling call
0929 14:11:48.308003 (+671636us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:11:48.308227 (+ 224us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:48.308527 (+ 300us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:11:48.388176 (+ 79649us) log.cc:842] Serialized 14 byte log entry
0929 14:11:48.441553 (+ 53377us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:48.441604 (+ 51us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:11:48.441630 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:48.441704 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:48.441805 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1804,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":164,"spinlock_wait_cycles":4970240,"thread_start_us":36760,"threads_started":1}
I20260929 14:11:48.442409 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.433362 6407 raft_consensus.cc:1067] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Signalling peer d886fb1eb0474d2d9f7410c4013c45e4 to start an election
I20260929 14:11:48.457909 6674 raft_consensus.cc:3037] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:48.432242 6751 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:48.471786 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.444937 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 110) took 1044 ms. Trace:
I20260929 14:11:48.445338 6587 consensus_queue.cc:1035] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260929 14:11:48.343431 6657 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:48.446955 6675 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.439540 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.476004 6563 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:48.493781 5195 raft_consensus.cc:1226] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:48.480319 6741 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.446329 6609 raft_consensus.cc:686] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:48.458307 6641 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:48.423207 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:48.482592 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.471884 6728 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:48.433033 6772 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:48.489800 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:48.473459 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:48.476823 6717 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:48.534451 6762 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:48.534332 6717 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:48.439440 5899 rpcz_store.cc:269] 0929 14:11:46.471510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:46.472966 (+ 1456us) service_pool.cc:227] Handling call
0929 14:11:48.313644 (+1840678us) spinlock_profiling.cc:243] Waited 1.65 ms on lock 0x7b5800027698. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:11:48.402645 (+ 89001us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3790592}
I20260929 14:11:48.536371 6717 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:48.496553 5200 raft_consensus.cc:1261] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:11:48.451859 6403 raft_consensus.cc:2781] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:48.470738 6718 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:48.471448 6650 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:48.488505 6751 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:48.530155 6769 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:48.491708 5870 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:48.541486 6741 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:48.532756 6731 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:48.549665 5346 raft_consensus.cc:2454] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 1.
I20260929 14:11:48.549890 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.557721 5351 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:11:48.501534 6675 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:48.531827 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:48.555056 6641 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:48.498785 6563 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.541168 6707 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:48.541437 6717 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:48.537549 5194 raft_consensus.cc:2454] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 1.
W20260929 14:11:48.552479 6776 log.cc:924] Time spent T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.114s user 0.001s sys 0.000s
I20260929 14:11:48.569391 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:48.569972 6770 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.571409 6657 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:48.564476 5197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 109) took 1181 ms. Trace:
I20260929 14:11:48.549228 6744 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:48.565757 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:48.567507 6778 log.cc:924] Time spent T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260929 14:11:48.343398 6658 rpcz_store.cc:276] 0929 14:11:43.544281 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:43.544438 (+ 157us) service_pool.cc:227] Handling call
0929 14:11:43.743269 (+198831us) spinlock_profiling.cc:243] Waited 1.28 ms on lock 0x7b2c000339d8. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda3450e11 00007fbda345fada 00007fbda345f763 00007fbda345f701 00007fbda4aa8b12 00007fbda4aa88ed 00007fbda4a73f84 00007fbda4a73743 00007fbda4a6bc1a 00007fbda99af6d9 00007fbda3099184
0929 14:11:43.770216 (+ 26947us) log.cc:842] Serialized 494 byte log entry
0929 14:11:44.217873 (+447657us) spinlock_profiling.cc:243] Waited 6.86 ms on lock 0x7b54000cb0a0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a06416 00007fbda49ffdaa 00007fbda49ff014 00007fbda49e7545 00007fbda49ec771 00007fbda49ec6f9 00007fbda49ec689 00007fbda49ec651 00007fbda49eb81d 00007fbdaada9204
0929 14:11:44.452977 (+235104us) spinlock_profiling.cc:243] Waited 1.35 ms on lock 0x7b18000e6920. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b12f96 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda49fea4f 00007fbda49e7545 00007fbda49ec771
0929 14:11:44.453623 (+ 646us) spinlock_profiling.cc:243] Waited 6.45 ms on lock 0x7b54000cb0a0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda49fea4f 00007fbda49e7545 00007fbda49ec771 00007fbda49ec6f9 00007fbda49ec689 00007fbda49ec651 00007fbda49eb81d 00007fbdaada9204 00007fbdaada9039 00007fbda1b456b6
0929 14:11:48.168012 (+3714389us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5648,"raft.run_cpu_time_us":8714,"raft.run_wall_time_us":739861,"spinlock_wait_cycles":38244352,"thread_start_us":196370,"threads_started":2,"wal-append.queue_time_us":202515,"wal-append.run_cpu_time_us":2419,"wal-append.run_wall_time_us":1008475}
I20260929 14:11:48.556078 6403 raft_consensus.cc:686] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:48.580417 5200 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:11:48.580873 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.572430 5197 rpcz_store.cc:276] 0929 14:11:47.383501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:47.899728 (+516227us) service_pool.cc:227] Handling call
0929 14:11:47.900058 (+ 330us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:47.964187 (+ 64129us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:47.964233 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:47.964300 (+ 67us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:11:47.964344 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:48.564278 (+599934us) spinlock_profiling.cc:243] Waited 63.9 ms on lock 0x7b540023c3d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:48.564314 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:48.564406 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":146921216}
I20260929 14:11:48.532833 6609 consensus_queue.cc:237] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.593781 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:48.563203 6662 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.563217 6769 raft_consensus.cc:2726] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:48.488268 5193 rpcz_store.cc:276] 0929 14:11:47.400996 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:48.115987 (+714991us) service_pool.cc:227] Handling call
0929 14:11:48.116317 (+ 330us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:48.130803 (+ 14486us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:48.135182 (+ 4379us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:48.137564 (+ 2382us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:11:48.150200 (+ 12636us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:48.436637 (+286437us) spinlock_profiling.cc:243] Waited 2.77 ms on lock 0x7b54002ce054. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:48.436690 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:48.444839 (+ 8149us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6378880}
I20260929 14:11:48.546774 6772 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.560312 6741 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.617897 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:48.621876 5197 raft_consensus.cc:2454] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:48.690783 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:48.568351 6756 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:48.690917 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 68803.9us
W20260929 14:11:48.581568 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 95251.8us
I20260929 14:11:48.694929 5198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52188 (request call id 1289) took 1668 ms. Trace:
I20260929 14:11:48.693524 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.696367 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:48.719035 5271 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50776 (request call id 1188) took 1824 ms (client timeout 1897 ms). Trace:
I20260929 14:11:48.698530 5873 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
W20260929 14:11:48.719729 5271 rpcz_store.cc:269] 0929 14:11:46.895039 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:46.895146 (+ 107us) service_pool.cc:227] Handling call
0929 14:11:48.718852 (+1823706us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b54003746d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:48.718975 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":989494016}
I20260929 14:11:48.720755 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.717978 5198 rpcz_store.cc:276] 0929 14:11:47.026234 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:47.051332 (+ 25098us) service_pool.cc:227] Handling call
0929 14:11:48.694776 (+1643444us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540023afd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:48.694872 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2779497344}
I20260929 14:11:48.721014 5164 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:48.721361 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 284327us
I20260929 14:11:48.721848 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.722256 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:48.722898 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.725270 5314 leader_election.cc:398] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3.
I20260929 14:11:48.718501 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:48.581307 6707 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.730248 5270 raft_consensus.cc:1226] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:48.730142 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.585258 6717 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:48.726552 6785 log.cc:924] Time spent T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20260929 14:11:48.726908 5267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:50768 (request call id 887) took 2411 ms. Trace:
I20260929 14:11:48.562058 6675 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.739166 6675 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.595501 6744 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:48.741662 6737 log.cc:924] Time spent T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 1.461s user 0.004s sys 0.004s
I20260929 14:11:48.742048 5268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1203) took 1151 ms. Trace:
I20260929 14:11:48.718892 6446 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:48.731246 5271 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:48.728289 6643 consensus_queue.cc:1035] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:48.726320 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:48.690418 6583 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:48.727286 6654 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:11:48.747030 5349 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:48.696810 6758 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:48.690351 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.736315 5267 rpcz_store.cc:276] 0929 14:11:46.309205 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:46.310673 (+ 1468us) service_pool.cc:227] Handling call
0929 14:11:48.720009 (+2409336us) spinlock_profiling.cc:243] Waited 646 ms on lock 0x7b54003741d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2af0 00007fbda4a75f2a 00007fbda99b2c6e 00007fbda30a9807 00007fbda30a9792 00007fbda30a96c1 00007fbda30a963c 00007fbda30a88f2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:48.720356 (+ 347us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1485414144}
I20260929 14:11:48.762467 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.696586 5083 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d653defbcc7e47ca89b35b1383e4e2c8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260929 14:11:48.763398 5083 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 147319us
I20260929 14:11:48.759351 6550 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:48.764492 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:48.765182 6550 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:48.765978 6563 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:48.767020 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:48.767704 5198 raft_consensus.cc:1799] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Handling vote request from an unknown peer c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:48.754388 5242 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:48.768970 5242 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:48.745401 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.770193 5242 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:48.730752 6403 consensus_queue.cc:237] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.759167 5350 raft_consensus.cc:1261] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:11:48.729295 6731 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:48.760497 6686 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:48.767222 6550 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:48.744589 5197 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:48.743155 6744 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.770761 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.757730 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.742451 5268 rpcz_store.cc:276] 0929 14:11:47.590124 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:47.608529 (+ 18405us) service_pool.cc:227] Handling call
0929 14:11:47.608861 (+ 332us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:47.903877 (+295016us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:47.904972 (+ 1095us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:47.914228 (+ 9256us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:47.917887 (+ 3659us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:48.741818 (+823931us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b54003232d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:48.741868 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:48.741991 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":526179712}
I20260929 14:11:48.732386 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:48.759146 6654 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:48.746452 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:48.782846 5271 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:48.777982 6650 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:48.785437 5198 raft_consensus.cc:2396] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260929 14:11:48.759971 6675 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:48.786389 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:48.788763 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.790010 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.791636 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.793192 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:48.772143 6446 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.761559 6718 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:48.795605 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:48.772020 6657 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:48.780403 6685 raft_consensus.cc:2781] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:48.778497 6619 raft_consensus.cc:2741] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260929 14:11:48.746150 5870 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:48.799115 6708 raft_consensus.cc:2726] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:48.798928 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:48.808004 5900 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45248 (request call id 226) took 2275 ms. Trace:
I20260929 14:11:48.813655 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52188 (request call id 1313) took 1475 ms. Trace:
I20260929 14:11:48.813985 6772 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.816349 6717 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:48.788311 6654 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:48.797847 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:48.761138 6583 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.827278 5267 raft_consensus.cc:2379] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:48.830647 5198 raft_consensus.cc:2379] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 3: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:48.830873 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.805609 6756 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.842950 5272 raft_consensus.cc:2354] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:11:48.804093 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.785019 6686 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.844197 5199 rpcz_store.cc:276] 0929 14:11:47.338175 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:47.338351 (+ 176us) service_pool.cc:227] Handling call
0929 14:11:48.813441 (+1475090us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b54001469d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:48.813580 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":518484736}
I20260929 14:11:48.827860 6642 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.797550 6675 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.873570 6585 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:48.771755 6758 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.832290 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.835974 6685 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:48.882974 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.900345 5344 raft_consensus.cc:3032] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:48.903927 5344 raft_consensus.cc:729] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:48.879004 6741 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.846490 6751 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.899159 5198 raft_consensus.cc:2932] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:48.911072 5197 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:48.924721 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.852749 5904 raft_consensus.cc:2454] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
W20260929 14:11:48.888517 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 156362us
I20260929 14:11:48.901808 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:48.933584 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 161933us
I20260929 14:11:48.837168 6657 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.923597 5344 consensus_queue.cc:260] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:48.938186 5344 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:48.929943 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.905692 6685 raft_consensus.cc:3037] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:48.943045 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.933779 5342 raft_consensus.cc:1226] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:48.934234 5345 raft_consensus.cc:1226] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260929 14:11:48.944532 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.945365 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.945667 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:48.946486 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.946671 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.947389 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:48.901405 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 131298us
I20260929 14:11:48.843982 5900 rpcz_store.cc:276] 0929 14:11:46.532993 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:46.533165 (+ 172us) service_pool.cc:227] Handling call
0929 14:11:47.593311 (+1060146us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:47.594083 (+ 772us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:48.807727 (+1213644us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400631ad0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6ca09 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:48.807904 (+ 177us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2437368192}
I20260929 14:11:48.947825 5267 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:48.948240 5242 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:48.937090 6674 raft_consensus.cc:2726] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:11:48.966164 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:48.994470 6776 log.cc:924] Time spent T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20260929 14:11:48.963780 6446 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.994783 6585 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:49.002897 6657 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:49.023579 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.956538 5167 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.000540 6751 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:48.964102 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:49.024042 6585 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:49.025318 5268 raft_consensus.cc:2379] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:49.001464 6583 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:48.948963 6619 raft_consensus.cc:2726] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:48.948912 5314 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:48.972965 6686 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:48.948693 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.970517 6758 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:49.001056 5196 consensus_meta.cc:287] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.551s user 0.009s sys 0.003s
I20260929 14:11:49.028955 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:48.972084 6675 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:48.957430 6732 consensus_meta.cc:287] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.505s user 0.003s sys 0.008s
I20260929 14:11:49.030958 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:49.031577 5200 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:11:49.032049 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.976030 5873 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.033201 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:49.021775 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:49.038704 5904 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:49.030146 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:49.039492 5273 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:49.036281 6657 raft_consensus.cc:2781] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:49.037148 5870 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:11:49.035636 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 86808.3us
I20260929 14:11:49.038399 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:49.023486 5272 raft_consensus.cc:2454] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:48.968523 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.994601 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:49.044165 5906 raft_consensus.cc:1799] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Handling vote request from an unknown peer c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:11:49.035503 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 79095.5us
I20260929 14:11:49.045009 5906 raft_consensus.cc:2396] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260929 14:11:49.046393 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:48.949385 5242 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.048910 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:49.043045 6728 raft_consensus.cc:2781] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:49.049412 5344 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:49.028710 6585 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:49.053282 5239 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:49.053891 5346 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
from {username='slave'} at 127.0.0.1:47258
I20260929 14:11:49.054457 5346 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:49.054889 5346 raft_consensus.cc:3037] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
W20260929 14:11:49.051453 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 102137us
W20260929 14:11:49.038412 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 62517.8us
I20260929 14:11:49.057550 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:49.059005 5267 raft_consensus.cc:2396] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260929 14:11:49.059245 5242 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.040717 6657 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:48.971495 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:49.092280 5242 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.063285 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:49.115459 5267 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:48.961753 5903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45232 (request call id 261) took 1007 ms. Trace:
I20260929 14:11:49.116308 5242 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.116578 5903 rpcz_store.cc:276] 0929 14:11:47.954769 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:47.954921 (+ 152us) service_pool.cc:227] Handling call
0929 14:11:48.961536 (+1006615us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b540029a254. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:48.961696 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":706338048}
I20260929 14:11:48.960871 5347 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 1637) took 1336 ms. Trace:
I20260929 14:11:49.117547 5242 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:48.975433 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:49.120151 5907 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:11:49.001065 6662 consensus_queue.cc:1035] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.092s
I20260929 14:11:49.029495 5196 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:11:49.036607 6583 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:49.091574 6657 raft_consensus.cc:3037] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:48.962116 5902 raft_consensus.cc:1226] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:49.132543 5242 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.135886 5195 raft_consensus.cc:1226] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:49.091293 6758 raft_consensus.cc:2781] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:49.141381 5350 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:49.093875 6769 raft_consensus.cc:2781] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260929 14:11:49.139434 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 47264.1us
I20260929 14:11:49.026252 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:49.157095 5165 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.026139 6654 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:49.055476 6619 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:49.093328 6446 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:49.046389 6685 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:49.173966 6685 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:49.158514 5165 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.177008 5165 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:49.091400 6736 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260929 14:11:49.177906 6732 raft_consensus.cc:2726] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:49.178406 6662 raft_consensus.cc:2726] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:49.179527 5165 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.180585 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:49.117914 5347 rpcz_store.cc:276] 0929 14:11:47.624310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:47.649919 (+ 25609us) service_pool.cc:227] Handling call
0929 14:11:48.718268 (+1068349us) spinlock_profiling.cc:243] Waited 5.63 ms on lock 0x7b5800183998. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:11:48.718482 (+ 214us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:48.724323 (+ 5841us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:48.724356 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:48.733149 (+ 8793us) log.cc:842] Serialized 26 byte log entry
0929 14:11:48.878953 (+145804us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:48.878994 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:48.879068 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:48.960546 (+ 81478us) raft_consensus.cc:1671] finished
0929 14:11:48.960567 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:48.960807 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5444,"raft.queue_time_us":7858,"spinlock_wait_cycles":16222208,"thread_start_us":118463,"threads_started":1,"wal-append.queue_time_us":180913}
I20260929 14:11:49.052860 6728 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:49.182621 6728 raft_consensus.cc:3037] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
W20260929 14:11:49.183290 5349 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1326) took 3912 ms (client timeout 3049 ms). Trace:
W20260929 14:11:49.183662 5349 rpcz_store.cc:269] 0929 14:11:45.270851 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:45.270999 (+ 148us) service_pool.cc:227] Handling call
0929 14:11:49.183092 (+3912093us) spinlock_profiling.cc:243] Waited 3.47 s on lock 0x7b54000659d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:49.183196 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7984002432}
I20260929 14:11:49.184633 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:49.136490 6789 raft_consensus.cc:2726] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:49.187254 6654 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:49.161516 6668 raft_consensus.cc:2726] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:49.154850 6583 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:49.130290 6792 raft_consensus.cc:2726] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260929 14:11:49.115852 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 78807.3us
I20260929 14:11:49.141398 6754 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:49.197121 5267 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:49.156241 6691 raft_consensus.cc:2781] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:49.155584 6769 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
W20260929 14:11:49.200292 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 98032.9us
I20260929 14:11:49.198993 5165 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:49.204926 5346 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:49.209456 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:49.211761 5314 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:49.223363 5167 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.223971 5873 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.168146 6758 raft_consensus.cc:686] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:49.196843 5900 raft_consensus.cc:1226] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5. Ops: []
I20260929 14:11:49.234108 6762 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:49.234710 6762 raft_consensus.cc:686] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 4 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:49.234804 6650 raft_consensus.cc:2726] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:49.235617 6762 consensus_queue.cc:237] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:49.236815 5165 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.238418 5165 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.238721 6642 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:49.226845 6622 raft_consensus.cc:2781] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:49.239557 6642 raft_consensus.cc:2781] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:49.240243 6675 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260929 14:11:49.242435 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 50709.7us
I20260929 14:11:49.243498 5269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1213) took 1306 ms. Trace:
I20260929 14:11:49.243912 5269 rpcz_store.cc:276] 0929 14:11:47.937388 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:48.071421 (+134033us) service_pool.cc:227] Handling call
0929 14:11:48.071662 (+ 241us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:48.266826 (+195164us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:48.266880 (+ 54us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:48.266921 (+ 41us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:48.266949 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:49.243177 (+976228us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b54002fdad4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:49.243242 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:49.243428 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6960,"raft.queue_time_us":3606,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":18661,"spinlock_wait_cycles":441037696}
I20260929 14:11:49.244844 6685 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:49.245474 6685 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:49.247525 5343 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47258 (request call id 1365) took 1113 ms. Trace:
I20260929 14:11:49.247758 6685 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:49.247982 5343 rpcz_store.cc:276] 0929 14:11:48.133916 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:48.134928 (+ 1012us) service_pool.cc:227] Handling call
0929 14:11:48.186076 (+ 51148us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:48.346307 (+160231us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:48.347354 (+ 1047us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:48.381348 (+ 33994us) log.cc:842] Serialized 26 byte log entry
0929 14:11:48.452765 (+ 71417us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:11:48.470918 (+ 18153us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:49.247071 (+776153us) spinlock_profiling.cc:243] Waited 99.8 ms on lock 0x7b540027d654. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:49.247127 (+ 56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:49.247180 (+ 53us) raft_consensus.cc:1671] finished
0929 14:11:49.247191 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:49.247456 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":5747,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":881,"spinlock_wait_cycles":229533568,"thread_start_us":56786,"threads_started":1,"wal-append.queue_time_us":107239}
I20260929 14:11:49.211133 6754 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:49.251940 6754 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:49.247543 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:49.254659 5870 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.254850 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:49.255105 5342 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 1647) took 1011 ms. Trace:
I20260929 14:11:49.255437 5342 rpcz_store.cc:276] 0929 14:11:48.243578 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:48.724896 (+481318us) service_pool.cc:227] Handling call
0929 14:11:48.740184 (+ 15288us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:48.943680 (+203496us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:49.254908 (+311228us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b54002d3cd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:49.255055 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":444713728}
I20260929 14:11:49.255882 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:49.256057 5870 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.206475 6744 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:49.256853 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:49.256982 6744 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:49.257439 6744 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:49.257825 5242 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.258586 5165 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.239627 6622 raft_consensus.cc:686] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:49.240195 6642 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:49.265560 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:49.225045 6769 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:49.222350 5193 raft_consensus.cc:1226] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:49.190099 5268 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:49.198906 6583 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:49.273180 5347 raft_consensus.cc:1226] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:49.273383 5316 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:49.273907 5242 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.277556 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:49.278162 5269 raft_consensus.cc:3032] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:49.278452 5269 raft_consensus.cc:729] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:49.279096 5269 consensus_queue.cc:260] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:49.285002 5198 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:11:49.285837 5198 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:11:49.286252 5198 raft_consensus.cc:3037] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:49.295486 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:49.214959 6691 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:49.297044 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:49.297699 5906 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:49.298806 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:49.301295 5269 raft_consensus.cc:3037] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:49.296149 5272 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:49.268998 6654 raft_consensus.cc:2726] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:49.272776 6642 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:11:49.304598 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 65183.8us
I20260929 14:11:49.317451 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:49.318166 5165 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.305135 6805 raft_consensus.cc:2726] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:49.324564 5269 raft_consensus.cc:2454] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:49.325866 5269 raft_consensus.cc:1226] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:49.326385 5201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1222) took 1080 ms. Trace:
I20260929 14:11:49.326751 5201 rpcz_store.cc:276] 0929 14:11:48.246079 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:48.414788 (+168709us) service_pool.cc:227] Handling call
0929 14:11:48.415149 (+ 361us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:48.485150 (+ 70001us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:48.485201 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:48.485274 (+ 73us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:48.485316 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:49.326187 (+840871us) spinlock_profiling.cc:243] Waited 69.8 ms on lock 0x7b540030b954. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:49.326231 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:49.326337 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":160428800}
I20260929 14:11:49.328210 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:49.308436 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:49.306324 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:49.301352 6691 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:49.313328 6732 raft_consensus.cc:2726] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:49.310137 6735 consensus_queue.cc:1035] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260929 14:11:49.344807 5268 raft_consensus.cc:1261] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:11:49.302276 6675 consensus_queue.cc:1046] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.039s
I20260929 14:11:49.345876 6675 raft_consensus.cc:3032] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:49.346189 6675 raft_consensus.cc:729] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:49.346971 6675 consensus_queue.cc:260] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:49.348210 6675 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:49.330550 6661 raft_consensus.cc:2726] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:11:49.355463 6800 log.cc:924] Time spent T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.122s user 0.002s sys 0.000s
I20260929 14:11:49.355147 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:49.324199 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:49.379261 5904 raft_consensus.cc:2379] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 2: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:11:49.380725 5348 raft_consensus.cc:1261] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:11:49.382748 5242 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.319545 5165 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.305648 6629 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260929 14:11:49.314206 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:49.277771 6685 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:49.397697 6685 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:49.399968 6685 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:49.315662 6657 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:49.315565 6758 consensus_queue.cc:237] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:49.401253 6728 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:49.330987 5314 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:49.291354 6407 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:49.333941 5271 raft_consensus.cc:1261] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:11:49.406570 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:49.334743 5201 raft_consensus.cc:1226] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:49.336042 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:49.408308 5200 raft_consensus.cc:2396] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260929 14:11:49.303843 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:49.301328 6789 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:11:49.302451 6562 raft_consensus.cc:2726] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:49.349167 6622 consensus_queue.cc:237] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:49.414577 5275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1211) took 1822 ms. Trace:
I20260929 14:11:49.412204 6789 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:49.415354 6789 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:49.425034 5270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1207) took 1833 ms. Trace:
I20260929 14:11:49.434020 5270 rpcz_store.cc:276] 0929 14:11:47.591170 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:47.721312 (+130142us) service_pool.cc:227] Handling call
0929 14:11:47.884437 (+163125us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:48.730704 (+846267us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:49.424747 (+694043us) spinlock_profiling.cc:243] Waited 846 ms on lock 0x7b5400392e54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:49.424961 (+ 214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1944704256}
W20260929 14:11:49.444484 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 61130.6us
W20260929 14:11:49.444941 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 62322.8us
I20260929 14:11:49.448117 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:49.415202 5275 rpcz_store.cc:276] 0929 14:11:47.592192 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:47.914641 (+322449us) service_pool.cc:227] Handling call
0929 14:11:47.914964 (+ 323us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:48.084519 (+169555us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:48.084556 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:48.099306 (+ 14750us) log.cc:842] Serialized 26 byte log entry
0929 14:11:48.826342 (+727036us) spinlock_profiling.cc:243] Waited 4.03 ms on lock 0x7b1800156940. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b27345 00007fbdaa95ea39 00007fbda1b448bb 00007fbda1b42cbf 00007fbda1b44e22 00007fbda46ee5bb 00007fbda46f4063 00007fbda46f4411 00007fbda4a415cf
0929 14:11:48.932592 (+106250us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:11:48.932653 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:49.414187 (+481534us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b54003223d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:49.414275 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:49.414337 (+ 62us) raft_consensus.cc:1671] finished
0929 14:11:49.414356 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:49.414512 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15117,"raft.queue_time_us":531,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":108,"spinlock_wait_cycles":398460672,"thread_start_us":105835,"threads_started":1,"wal-append.queue_time_us":874977}
I20260929 14:11:49.454102 5275 raft_consensus.cc:1226] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
W20260929 14:11:49.385013 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 65629.2us
W20260929 14:11:49.390141 6803 log.cc:924] Time spent T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29: Append to log took a long time: real 0.074s user 0.000s sys 0.001s
I20260929 14:11:49.353988 5870 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:49.337532 6718 consensus_queue.cc:1035] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.040s
I20260929 14:11:49.394896 5196 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
W20260929 14:11:49.317037 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 165186us
I20260929 14:11:49.402067 6657 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:49.483929 6728 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:49.484656 6789 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:49.410120 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 79241.6us
I20260929 14:11:49.488279 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:49.488865 5906 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:49.491395 6789 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:49.467540 5165 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:49.494109 5165 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:49.494797 6741 raft_consensus.cc:2726] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:49.495357 6741 raft_consensus.cc:2726] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:49.495684 5165 leader_election.cc:398] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:11:49.409195 6661 raft_consensus.cc:2726] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:49.435822 6476 raft_consensus.cc:2781] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election won for term 2
W20260929 14:11:49.469614 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 150190us
I20260929 14:11:49.473294 5198 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:49.499297 6583 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:49.500030 5198 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:49.312129 5346 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:49.500752 5349 raft_consensus.cc:1226] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260929 14:11:49.501098 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:49.501971 5907 raft_consensus.cc:2379] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 2: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:11:49.409862 6407 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:49.502696 6407 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:49.496912 5165 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:49.492389 6635 consensus_queue.cc:1046] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:49.496867 5906 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:11:49.505349 5165 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.460326 6446 raft_consensus.cc:3037] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:49.516719 5316 leader_election.cc:398] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3.
I20260929 14:11:49.520390 6585 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260929 14:11:49.520702 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 178414us
I20260929 14:11:49.522415 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:49.522812 6635 consensus_queue.cc:1046] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:49.523931 6635 consensus_queue.cc:1035] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:49.528034 6403 consensus_queue.cc:1046] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:49.508191 6657 raft_consensus.cc:2726] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:49.529286 6641 raft_consensus.cc:3032] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:49.529881 6641 raft_consensus.cc:729] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:49.530385 6718 raft_consensus.cc:3032] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:49.530791 6718 raft_consensus.cc:729] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:49.530834 6641 consensus_queue.cc:260] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:49.532435 6641 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:49.531942 6718 consensus_queue.cc:260] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:49.534034 6733 raft_consensus.cc:3032] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:49.534495 6733 raft_consensus.cc:729] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:49.509917 5165 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.511467 5242 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:49.537412 5165 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.507229 5239 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:49.559098 6744 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:49.463969 5270 raft_consensus.cc:1261] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:11:49.518317 6741 raft_consensus.cc:2781] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:49.563916 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:49.513154 5271 raft_consensus.cc:1261] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:11:49.544791 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1236) took 1242 ms. Trace:
W20260929 14:11:49.571260 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 64125.2us
I20260929 14:11:49.571412 5199 rpcz_store.cc:276] 0929 14:11:48.301911 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:48.890142 (+588231us) service_pool.cc:227] Handling call
0929 14:11:48.973700 (+ 83558us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:48.973862 (+ 162us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:48.973884 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:48.973918 (+ 34us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:11:48.973948 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:48.973997 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:49.544541 (+570544us) spinlock_profiling.cc:243] Waited 83.2 ms on lock 0x7b54003dbb50. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6ca09 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:49.544709 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":191427968}
I20260929 14:11:49.549299 6619 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:11:49.518654 6476 raft_consensus.cc:686] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:49.520912 6585 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:49.553771 6733 consensus_queue.cc:260] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:49.552814 6707 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:49.573127 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:49.460744 6792 consensus_queue.cc:1035] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.080s
W20260929 14:11:49.612708 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 45296.8us
I20260929 14:11:49.620203 5165 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:49.621829 5165 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.627717 5165 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.585919 6742 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:49.603979 6718 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:49.574327 6662 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:49.574597 6741 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:49.661092 6732 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:49.644742 6619 raft_consensus.cc:686] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:49.586769 6657 raft_consensus.cc:2726] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:49.662721 6476 consensus_queue.cc:237] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:49.706239 6619 consensus_queue.cc:237] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:49.564925 5273 consensus_meta.cc:287] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.525s user 0.007s sys 0.003s
I20260929 14:11:49.709125 5273 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:49.709879 5273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1596) took 1262 ms. Trace:
I20260929 14:11:49.623365 6650 raft_consensus.cc:2726] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:49.667460 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:11:49.710664 5274 raft_consensus.cc:1226] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:49.644794 6585 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:49.646893 6668 raft_consensus.cc:2726] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:49.585538 6609 consensus_queue.cc:1046] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.080s
I20260929 14:11:49.586165 6736 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:49.665410 6742 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:49.710327 5273 rpcz_store.cc:276] 0929 14:11:48.447726 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:49.029895 (+582169us) service_pool.cc:227] Handling call
0929 14:11:49.709803 (+679908us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:49.720717 6741 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:11:49.742903 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 236617us
W20260929 14:11:49.747041 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 227858us
I20260929 14:11:49.754791 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:49.757839 5274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1222) took 1506 ms. Trace:
I20260929 14:11:49.758340 5274 rpcz_store.cc:276] 0929 14:11:48.251467 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:49.155154 (+903687us) service_pool.cc:227] Handling call
0929 14:11:49.710395 (+555241us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:49.711155 (+ 760us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:49.757594 (+ 46439us) spinlock_profiling.cc:243] Waited 525 ms on lock 0x7b540007d850. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6ca09 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:49.757768 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1206349568}
I20260929 14:11:49.764254 5347 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47258 (request call id 1384) took 1040 ms. Trace:
I20260929 14:11:49.767066 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1229) took 1519 ms. Trace:
I20260929 14:11:49.767539 5193 rpcz_store.cc:276] 0929 14:11:48.247965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:48.814908 (+566943us) service_pool.cc:227] Handling call
0929 14:11:49.190679 (+375771us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:49.302197 (+111518us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:49.766819 (+464622us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b54001455d0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6ca09 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:49.766978 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":859135744}
I20260929 14:11:49.771209 5902 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45248 (request call id 231) took 1514 ms. Trace:
I20260929 14:11:49.771679 5902 rpcz_store.cc:276] 0929 14:11:48.256325 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:48.259793 (+ 3468us) service_pool.cc:227] Handling call
0929 14:11:48.961905 (+702112us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:49.158709 (+196804us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:49.770955 (+612246us) spinlock_profiling.cc:243] Waited 690 ms on lock 0x7b540029a250. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6ca09 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:49.771137 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1587675904}
I20260929 14:11:49.764731 5347 rpcz_store.cc:276] 0929 14:11:48.723672 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:49.189711 (+466039us) service_pool.cc:227] Handling call
0929 14:11:49.190034 (+ 323us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:49.277412 (+ 87378us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:49.763965 (+486553us) spinlock_profiling.cc:243] Waited 82.9 ms on lock 0x7b5400258fd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:49.764184 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":190750592}
I20260929 14:11:49.816751 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:49.817674 5195 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:49.739992 6742 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:49.722406 6754 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:49.827517 5316 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:49.827914 5269 raft_consensus.cc:1226] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:49.665051 6407 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:49.848284 5275 raft_consensus.cc:1226] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:49.820328 6654 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:49.739180 6736 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:49.741087 6769 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:49.895186 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 267546us
I20260929 14:11:49.896487 5165 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.898221 5165 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.900619 5165 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.854884 6744 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:49.913249 5907 raft_consensus.cc:1226] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:49.913364 5165 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:49.906207 6808 log.cc:924] Time spent T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
W20260929 14:11:49.899776 6796 log.cc:924] Time spent T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260929 14:11:49.839346 6551 consensus_queue.cc:1035] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.067s
I20260929 14:11:49.839538 6629 consensus_queue.cc:1035] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260929 14:11:49.926877 5347 raft_consensus.cc:1261] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:49.928136 5165 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.898543 6659 raft_consensus.cc:2726] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:49.898372 6567 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:49.906646 6668 raft_consensus.cc:2726] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:49.942466 5276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1217) took 1840 ms. Trace:
I20260929 14:11:49.942684 6769 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:49.938005 6567 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:49.942864 5276 rpcz_store.cc:276] 0929 14:11:48.102168 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:48.416576 (+314408us) service_pool.cc:227] Handling call
0929 14:11:48.416928 (+ 352us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:48.425216 (+ 8288us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:48.425268 (+ 52us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:48.431888 (+ 6620us) log.cc:842] Serialized 26 byte log entry
0929 14:11:48.740527 (+308639us) spinlock_profiling.cc:243] Waited 8.65 ms on lock 0x7b1800156640. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda46fc82c 00007fbda1b279dc 00007fbdaa95ea39 00007fbda1b448bb 00007fbda1b42cbf 00007fbda1b44e22 00007fbda46ee5bb 00007fbda46f4063 00007fbda46f4411
0929 14:11:48.740662 (+ 135us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:11:48.740695 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:48.740756 (+ 61us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:49.933301 (+1192545us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x7b1800156640. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b45638 00007fbda1b48b11 00007fbda1b48ac9 00007fbda1b48a59 00007fbda1b48a21 00007fbda1b47d1d 00007fbdaada9204 00007fbdaada9039 00007fbda1b27dc6
0929 14:11:49.942284 (+ 8983us) raft_consensus.cc:1671] finished
0929 14:11:49.942304 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:49.942412 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21126,"raft.queue_time_us":8047,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":12788,"spinlock_wait_cycles":27266816,"thread_start_us":14663,"threads_started":1,"wal-append.queue_time_us":14571}
I20260929 14:11:49.944067 6585 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:49.929167 6642 consensus_meta.cc:287] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.588s user 0.004s sys 0.006s
I20260929 14:11:49.945408 5906 raft_consensus.cc:1226] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5. Ops: []
I20260929 14:11:49.945806 5165 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:49.946053 6585 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:49.947225 6657 raft_consensus.cc:2781] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:49.947600 6657 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:49.947992 6657 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:49.948350 6732 consensus_queue.cc:1035] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:49.895291 6654 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:49.921736 6731 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:49.933743 6662 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:49.958887 6736 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:49.959400 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:49.953090 6568 consensus_queue.cc:1035] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260929 14:11:49.964493 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:49.964675 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:49.966153 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:50.031772 5272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1227) took 1540 ms. Trace:
I20260929 14:11:50.032208 5272 rpcz_store.cc:276] 0929 14:11:48.491271 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:49.338884 (+847613us) service_pool.cc:227] Handling call
0929 14:11:49.380454 (+ 41570us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:49.381968 (+ 1514us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:49.381997 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:49.382391 (+ 394us) log.cc:842] Serialized 26 byte log entry
0929 14:11:49.891953 (+509562us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:49.891998 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:49.892060 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:50.031524 (+139464us) raft_consensus.cc:1671] finished
0929 14:11:50.031550 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:50.031702 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1046,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":107,"thread_start_us":470918,"threads_started":1,"wal-append.queue_time_us":646245}
I20260929 14:11:50.033285 5907 raft_consensus.cc:1226] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260929 14:11:50.033820 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:11:50.019471 5084 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 190298us
W20260929 14:11:50.022219 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 194773us
W20260929 14:11:50.023641 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 60428.7us
I20260929 14:11:50.035831 5267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50770 (request call id 1584) took 1615 ms. Trace:
I20260929 14:11:50.035959 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:11:49.987018 6809 log.cc:924] Time spent T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.245s user 0.001s sys 0.000s
I20260929 14:11:50.036427 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:50.041420 5344 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:50.039748 5316 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:49.987257 6659 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:49.990971 6567 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:49.912425 6812 consensus_queue.cc:1046] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.130s
W20260929 14:11:50.057236 6815 log.cc:924] Time spent T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
I20260929 14:11:50.049052 5239 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:49.992980 6662 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:49.990454 6744 consensus_queue.cc:1035] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.059s
I20260929 14:11:50.044533 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:50.072072 6446 consensus_meta.cc:287] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.536s user 0.009s sys 0.000s
I20260929 14:11:50.072284 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:50.072666 6446 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:50.058312 6643 consensus_queue.cc:1046] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260929 14:11:50.073738 6654 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:50.063773 6659 raft_consensus.cc:686] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:49.945427 6642 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.075387 6659 consensus_queue.cc:237] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.076237 6642 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:50.077066 5276 raft_consensus.cc:1226] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:50.077582 6754 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:50.036186 5267 rpcz_store.cc:276] 0929 14:11:48.420879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:49.270511 (+849632us) service_pool.cc:227] Handling call
0929 14:11:49.271031 (+ 520us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:49.271269 (+ 238us) raft_consensus.cc:1520] Early marking committed up to index 2
0929 14:11:49.303249 (+ 31980us) log.cc:842] Serialized 14 byte log entry
0929 14:11:49.956068 (+652819us) spinlock_profiling.cc:243] Waited 8.41 ms on lock 0x7b18004161a0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b27345 00007fbdaa95ea39 00007fbda1b448bb 00007fbda1b42cbf 00007fbda1b44e22 00007fbda46ee5bb 00007fbda46f4063 00007fbda46f457b 00007fbda4a7788a
0929 14:11:50.035064 (+ 78996us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:50.035129 (+ 65us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:50.035280 (+ 151us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:50.035342 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:50.035778 (+ 436us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3823,"prepare.queue_time_us":29802,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":12650,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":102,"spinlock_wait_cycles":19348736,"thread_start_us":67497,"threads_started":2}
I20260929 14:11:50.078357 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:50.079075 5193 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:50.079872 5316 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:50.080729 6735 raft_consensus.cc:2726] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:11:50.080955 6812 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:50.081341 6812 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:50.081748 6812 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:50.081451 6657 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.064646 6732 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:50.035665 6685 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:50.039465 5273 raft_consensus.cc:1226] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 LEADER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
I20260929 14:11:50.063395 6662 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:50.065955 6567 raft_consensus.cc:2781] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:50.022249 6792 consensus_queue.cc:1046] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.042s
I20260929 14:11:50.043531 6822 raft_consensus.cc:2726] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:50.059022 6717 consensus_queue.cc:1046] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
W20260929 14:11:50.054611 6691 consensus_meta.cc:287] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.701s user 0.006s sys 0.005s
I20260929 14:11:50.120889 6691 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.111476 5870 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:50.123000 6691 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:50.123097 6762 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:50.092314 6731 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:50.102191 6732 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.098256 5192 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:50.103910 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:50.125252 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:50.125455 5195 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:50.126008 5192 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:50.126962 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:50.106609 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:50.127934 5347 raft_consensus.cc:3032] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:50.128309 5347 raft_consensus.cc:729] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:50.128737 6834 raft_consensus.cc:3032] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:50.129135 6834 raft_consensus.cc:729] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:50.099594 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
W20260929 14:11:50.063460 6641 consensus_meta.cc:287] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.530s user 0.010s sys 0.003s
I20260929 14:11:50.134716 5902 raft_consensus.cc:1226] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260929 14:11:50.130187 5268 raft_consensus.cc:1226] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:50.129297 5347 consensus_queue.cc:260] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:50.101764 6685 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.136118 5347 raft_consensus.cc:3037] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:50.126899 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:50.137420 6685 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:50.103734 6567 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:50.138007 6567 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:50.107563 6675 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:50.138906 6675 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.140854 6675 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:50.130501 5907 raft_consensus.cc:3037] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:50.115833 5316 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:50.123916 6731 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.131758 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:50.133055 5873 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:50.147957 5316 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:50.148936 5316 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:11:50.145429 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 317140us
I20260929 14:11:50.089800 6742 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.155169 5903 raft_consensus.cc:1226] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:50.156268 5316 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:50.156788 5903 raft_consensus.cc:1226] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:50.128324 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:50.152026 6835 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:11:50.160495 5195 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:50.152667 6641 raft_consensus.cc:2781] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:50.164103 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:50.152535 6728 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:11:50.152176 6811 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260929 14:11:50.146612 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 44497.3us
I20260929 14:11:50.165063 6717 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:11:50.174942 6717 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:50.175418 6717 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:50.159327 6476 consensus_queue.cc:1046] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260929 14:11:50.177785 6812 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.167200 6641 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:50.179062 6641 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:50.179844 6812 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:50.180091 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:50.180439 6840 consensus_queue.cc:1046] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260929 14:11:50.181823 6840 raft_consensus.cc:3032] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:50.182178 6840 raft_consensus.cc:729] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:50.183139 6840 consensus_queue.cc:260] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:50.184659 6840 raft_consensus.cc:3037] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:50.157963 6551 consensus_queue.cc:1046] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20260929 14:11:50.189373 6476 raft_consensus.cc:3032] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:11:50.189587 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:50.189918 6476 raft_consensus.cc:729] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:50.190989 6476 consensus_queue.cc:260] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:50.193032 6657 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:50.198251 5873 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:50.201584 5192 raft_consensus.cc:1261] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:11:50.203078 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:50.203478 5873 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:50.204290 6762 raft_consensus.cc:2726] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:11:50.204634 6728 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:50.205412 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:50.157830 6834 consensus_queue.cc:260] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:50.207617 5347 raft_consensus.cc:2454] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:50.201602 6731 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:50.174188 6732 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:50.211794 6551 consensus_queue.cc:1035] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260929 14:11:50.216822 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:50.161341 6741 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.225085 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:50.224177 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:50.228731 6789 raft_consensus.cc:3032] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:11:50.229111 6789 raft_consensus.cc:729] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:50.232257 5907 raft_consensus.cc:2454] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:50.232481 5873 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:50.192644 6476 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:50.231207 6567 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.242128 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:50.231555 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:50.246693 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:50.247323 5347 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:50.240693 6728 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:50.256597 5164 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:50.260202 5164 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:50.260627 6567 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:50.231215 6717 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.261797 5350 raft_consensus.cc:2354] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5.
I20260929 14:11:50.264084 6741 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:50.259281 6844 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:50.265056 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:50.265712 5194 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:50.265733 5907 raft_consensus.cc:1261] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:50.267810 6717 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:50.251351 6659 consensus_queue.cc:1046] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260929 14:11:50.268450 6717 consensus_queue.cc:1035] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:11:50.269047 6659 raft_consensus.cc:3032] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:50.269428 6659 raft_consensus.cc:729] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:50.270452 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:50.270226 6659 consensus_queue.cc:260] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.271438 6659 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:50.271677 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:50.247218 6742 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:50.275849 6731 raft_consensus.cc:2781] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:50.276379 6731 raft_consensus.cc:686] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:50.277267 6731 consensus_queue.cc:237] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:50.288002 5164 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:50.252375 6789 consensus_queue.cc:260] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:50.289553 6732 raft_consensus.cc:2781] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:50.289940 6789 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:50.289976 6732 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:50.290946 6732 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:50.272332 5273 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:50.294653 5314 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:50.295660 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:50.297734 6840 raft_consensus.cc:2726] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:50.299368 5192 raft_consensus.cc:1226] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260929 14:11:50.300901 5192 raft_consensus.cc:1261] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260929 14:11:50.301271 5192 pending_rounds.cc:76] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd: Aborting all ops after (but not including) 1
I20260929 14:11:50.301503 5192 pending_rounds.cc:97] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260929 14:11:50.301786 5192 raft_consensus.cc:2866] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:11:50.302524 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:50.303416 5270 raft_consensus.cc:2354] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4.
I20260929 14:11:50.303416 6717 consensus_queue.cc:1035] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:50.304908 6732 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.307051 5192 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260929 14:11:50.307600 6732 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:50.311797 5275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1237) took 1267 ms. Trace:
I20260929 14:11:50.312332 5275 rpcz_store.cc:276] 0929 14:11:49.044668 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:49.456256 (+411588us) service_pool.cc:227] Handling call
0929 14:11:49.468487 (+ 12231us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:49.855796 (+387309us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:50.311569 (+455773us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b5400134554. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:50.311735 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":873021184}
I20260929 14:11:50.314286 6728 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 14:11:50.315943 6662 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.317514 5199 raft_consensus.cc:1226] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260929 14:11:50.318813 6662 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:50.321759 5349 raft_consensus.cc:3032] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:50.322101 5349 raft_consensus.cc:729] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:50.324285 6717 consensus_queue.cc:1046] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260929 14:11:50.324879 6662 consensus_queue.cc:1046] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260929 14:11:50.353785 5314 leader_election.cc:398] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4.
I20260929 14:11:50.354257 5314 leader_election.cc:401] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:11:50.355043 5275 raft_consensus.cc:1226] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:50.358284 5201 raft_consensus.cc:1261] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:11:50.359118 6844 raft_consensus.cc:3032] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:11:50.359499 6844 raft_consensus.cc:729] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:50.324545 5349 consensus_queue.cc:260] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:50.362991 5349 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:50.365815 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:50.369429 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:50.370955 6551 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 4
I20260929 14:11:50.374967 6641 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:50.362569 6403 raft_consensus.cc:3032] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:50.371824 6844 consensus_queue.cc:260] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:50.385466 6731 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:50.391345 5201 raft_consensus.cc:1226] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:50.390697 6731 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.394548 6731 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:50.378371 6562 consensus_queue.cc:1046] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260929 14:11:50.395615 6562 raft_consensus.cc:3032] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:50.395956 6562 raft_consensus.cc:729] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:50.386541 6403 raft_consensus.cc:729] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:50.391480 6736 consensus_queue.cc:1046] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.007s
I20260929 14:11:50.398356 6641 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:50.398680 6736 raft_consensus.cc:3032] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:50.399020 6736 raft_consensus.cc:729] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:50.400043 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:50.399808 6736 consensus_queue.cc:260] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:50.412472 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:50.397686 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:50.403414 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:50.426378 5314 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:50.428380 5345 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 1656) took 1927 ms. Trace:
I20260929 14:11:50.430902 5345 rpcz_store.cc:276] 0929 14:11:48.500767 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:48.947490 (+446723us) service_pool.cc:227] Handling call
0929 14:11:49.325036 (+377546us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:49.723542 (+398506us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:49.723590 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:49.731554 (+ 7964us) log.cc:842] Serialized 26 byte log entry
0929 14:11:50.011387 (+279833us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:11:50.011448 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:50.011564 (+ 116us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:50.124905 (+113341us) raft_consensus.cc:1671] finished
0929 14:11:50.124932 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:50.421945 (+297013us) spinlock_profiling.cc:243] Waited 377 ms on lock 0x7b54003a98d0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6ca09 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:50.428305 (+ 6360us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1050,"raft.queue_time_us":24721,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":15985,"spinlock_wait_cycles":867598592,"thread_start_us":673945,"threads_started":2,"wal-append.queue_time_us":387522}
W20260929 14:11:50.431403 5084 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet f84af62485a4405f887bb8675221641e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20260929 14:11:50.400974 6736 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:50.396708 6562 consensus_queue.cc:260] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.401675 6844 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:50.417083 6551 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4.
I20260929 14:11:50.429484 6476 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:11:50.410982 5273 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:11:50.443871 5343 raft_consensus.cc:2354] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:11:50.448418 6583 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:50.449753 6562 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:50.445187 6840 consensus_queue.cc:1046] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260929 14:11:50.455143 6836 raft_consensus.cc:3032] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:50.455560 6836 raft_consensus.cc:729] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:50.457600 5349 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260929 14:11:50.459126 6735 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260929 14:11:50.456694 6836 consensus_queue.cc:260] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.464978 6836 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:50.680891 6407 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:50.714152 5164 leader_election.cc:398] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:11:50.714543 5164 leader_election.cc:401] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260929 14:11:50.716318 5194 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:11:50.722430 5276 raft_consensus.cc:1226] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:50.730157 6407 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:50.730640 6407 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:50.732594 6407 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:50.750602 6826 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:51.510783 6741 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:51.523876 6550 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:51.727334 5130 debug-util.cc:398] Leaking SignalData structure 0x7b08000333e0 after lost signal to thread 4044
I20260929 14:11:51.929852 6407 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.170626 6407 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.276679 6731 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.276918 6550 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.277712 6731 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.276839 6728 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.279971 6733 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:52.285444 6728 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.287124 6407 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.287850 6407 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.288043 6789 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.288582 6728 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.288597 6407 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.289995 6792 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.290766 6792 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.291778 6407 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.292335 6407 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.289281 6731 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.288925 6789 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.264096 6741 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.294165 6792 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.293121 6407 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.304925 6789 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.296787 6550 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.305872 6407 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.321736 6741 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:52.336298 6826 consensus_meta.cc:287] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 1.585s user 0.007s sys 0.013s
I20260929 14:11:52.336922 6826 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:11:52.341899 6733 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.342918 6733 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.345968 6733 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:52.352139 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 52839.9us
I20260929 14:11:52.353422 6826 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.353678 6674 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.354265 6826 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.354550 6674 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.356494 6826 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.357199 6674 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.359176 6733 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.359690 6733 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.361304 6733 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.361824 6733 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:52.362323 6741 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:52.362569 6733 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.363036 6550 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.363206 6741 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.363869 6550 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.364836 6733 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.365850 6550 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.365863 6741 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.366168 6844 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.366994 6844 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.369050 6844 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.362409 6826 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.370204 6826 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.373075 6826 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.384330 6867 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.385169 6867 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.387709 6867 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.388238 6867 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.388854 6867 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.392302 6867 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.392786 6867 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.393558 6867 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.396064 6867 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.396659 6867 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.397382 6867 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.399734 6867 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.400300 6867 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:52.401142 6867 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.403669 6867 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.409622 6867 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.410055 6736 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:52.410209 6867 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.411026 6736 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.412446 6867 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.412952 6867 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:52.413628 6867 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.415907 6867 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.416424 6867 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:52.423094 6844 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.423353 6550 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:52.423946 6844 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.424755 6403 consensus_queue.cc:260] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.426909 6650 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.427050 6403 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:52.427143 6731 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.427956 6650 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.429023 6731 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.439718 6731 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.437153 6834 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:52.446883 6844 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.447835 5269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1226) took 3977 ms. Trace:
I20260929 14:11:52.424427 6550 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.429800 6728 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.450930 6550 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.450623 6728 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.453482 5268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50770 (request call id 1604) took 3890 ms. Trace:
I20260929 14:11:52.453876 5268 rpcz_store.cc:276] 0929 14:11:48.562748 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:49.348504 (+785756us) service_pool.cc:227] Handling call
0929 14:11:49.348797 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:50.135415 (+786618us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:52.452994 (+2317579us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b54003228d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:52.453331 (+ 337us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1796567296}
I20260929 14:11:52.454274 6728 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.458541 6728 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:52.459429 6728 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.448227 5269 rpcz_store.cc:276] 0929 14:11:48.470475 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:49.326549 (+856074us) service_pool.cc:227] Handling call
0929 14:11:49.326842 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:49.831900 (+505058us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:52.447397 (+2615497us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x7b54001095d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:52.447658 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1151982848}
I20260929 14:11:52.417030 6867 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.468283 6707 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.468972 6707 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.471011 6707 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.471153 6867 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.434242 5270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1252) took 2215 ms. Trace:
I20260929 14:11:52.504977 5270 rpcz_store.cc:276] 0929 14:11:50.218686 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:50.304902 (+ 86216us) service_pool.cc:227] Handling call
0929 14:11:50.305170 (+ 268us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:50.329657 (+ 24487us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:50.329701 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:50.353522 (+ 23821us) log.cc:842] Serialized 26 byte log entry
0929 14:11:52.404332 (+2050810us) spinlock_profiling.cc:243] Waited 2.23 ms on lock 0x7b18000b6f80. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b27345 00007fbdaa95ea39 00007fbda1b448bb 00007fbda1b42cbf 00007fbda1b44e22 00007fbda46ee5bb 00007fbda46f4063 00007fbda46f4411 00007fbda4a415cf
0929 14:11:52.433135 (+ 28803us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:52.433240 (+ 105us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:52.433408 (+ 168us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:52.433501 (+ 93us) raft_consensus.cc:1671] finished
0929 14:11:52.433525 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:52.433899 (+ 374us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":6750,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":17532,"spinlock_wait_cycles":5120768,"thread_start_us":28367,"threads_started":1,"wal-append.queue_time_us":2048765}
I20260929 14:11:52.435266 6736 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.448570 6870 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.508711 6875 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.508545 6870 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.509688 6875 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.432411 6650 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.472389 5349 raft_consensus.cc:1261] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:52.474140 6654 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:52.474645 6871 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.475533 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:52.513087 5198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1341) took 2458 ms. Trace:
I20260929 14:11:52.477807 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.513947 6654 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.514444 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.522172 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.523133 5348 raft_consensus.cc:2379] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 2: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:52.523363 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.523435 6654 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.523948 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.524557 6654 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.476827 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.524827 5348 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:11:52.486498 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.515372 6870 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.492205 5272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1250) took 2274 ms. Trace:
W20260929 14:11:52.494837 5350 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47252 (request call id 1680) took 2334 ms (client timeout 1529 ms). Trace:
W20260929 14:11:52.495899 5195 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52182 (request call id 1253) took 2398 ms (client timeout 1780 ms). Trace:
I20260929 14:11:52.477975 6728 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.525197 5347 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:52.526005 6870 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.526155 6728 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.502835 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:52.526872 6870 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.527031 6728 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.518122 6871 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.519227 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.530431 5164 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:52.531167 6871 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.531448 6662 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:11:52.531914 6662 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:52.519786 5342 raft_consensus.cc:2354] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5.
I20260929 14:11:52.497804 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.532884 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:52.514935 6650 consensus_queue.cc:1035] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
I20260929 14:11:52.533628 5197 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:52.525686 5272 rpcz_store.cc:276] 0929 14:11:50.218045 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:52.491078 (+2273033us) service_pool.cc:227] Handling call
0929 14:11:52.491531 (+ 453us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:52.491759 (+ 228us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:52.491786 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:52.491837 (+ 51us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:52.491865 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:52.491919 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:52.492043 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:52.534150 5274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 110) took 2461 ms. Trace:
W20260929 14:11:52.534324 5197 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52188 (request call id 1343) took 2479 ms (client timeout 1544 ms). Trace:
I20260929 14:11:52.534530 5274 rpcz_store.cc:276] 0929 14:11:50.073046 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:52.532844 (+2459798us) service_pool.cc:227] Handling call
0929 14:11:52.533392 (+ 548us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:52.533721 (+ 329us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:52.533760 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:52.533814 (+ 54us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:11:52.533845 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:52.533895 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:52.534002 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260929 14:11:52.504026 5193 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.144s user 0.022s sys 0.046s
I20260929 14:11:52.535879 6870 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:11:52.536306 5193 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.176s user 0.023s sys 0.047s
I20260929 14:11:52.527506 5201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1265) took 2136 ms. Trace:
I20260929 14:11:52.536700 5201 rpcz_store.cc:276] 0929 14:11:50.390520 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:50.390845 (+ 325us) service_pool.cc:227] Handling call
0929 14:11:50.391155 (+ 310us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:52.527036 (+2135881us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:52.527359 (+ 323us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:52.531176 5345 raft_consensus.cc:2396] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260929 14:11:52.537446 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 134) took 2221 ms. Trace:
I20260929 14:11:52.505282 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.537962 6674 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:52.538807 6870 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.538731 6674 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.535598 5351 raft_consensus.cc:2379] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:52.539573 6870 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.525558 5343 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
W20260929 14:11:52.525818 5350 rpcz_store.cc:269] 0929 14:11:50.160774 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:50.163879 (+ 3105us) service_pool.cc:227] Handling call
0929 14:11:52.494460 (+2330581us) spinlock_profiling.cc:243] Waited 95.3 ms on lock 0x7b5400386154. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:52.494663 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":219153152}
I20260929 14:11:52.541839 6870 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.527648 5196 raft_consensus.cc:2396] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260929 14:11:52.542459 6870 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.541769 5164 leader_election.cc:398] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5.
I20260929 14:11:52.537886 5193 rpcz_store.cc:276] 0929 14:11:50.316418 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:50.358690 (+ 42272us) service_pool.cc:227] Handling call
0929 14:11:50.359030 (+ 340us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:50.359357 (+ 327us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:50.359399 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:50.359851 (+ 452us) log.cc:842] Serialized 26 byte log entry
0929 14:11:52.536898 (+2177047us) raft_consensus.cc:1631] Marking committed up to 6
0929 14:11:52.536973 (+ 75us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:52.537097 (+ 124us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:52.537158 (+ 61us) raft_consensus.cc:1671] finished
0929 14:11:52.537183 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:52.537366 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1609,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":4170,"thread_start_us":2176610,"threads_started":1,"wal-append.queue_time_us":99242,"wal-append.run_cpu_time_us":1965,"wal-append.run_wall_time_us":1028071}
W20260929 14:11:52.543094 5196 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52188 (request call id 1335) took 2579 ms (client timeout 1612 ms). Trace:
W20260929 14:11:52.534729 5197 rpcz_store.cc:269] 0929 14:11:50.054575 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:50.057313 (+ 2738us) service_pool.cc:227] Handling call
0929 14:11:52.532511 (+2475198us) spinlock_profiling.cc:243] Waited 2.41 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:11:52.534270 (+ 1759us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5538560}
I20260929 14:11:52.543365 6845 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:11:52.543622 5196 rpcz_store.cc:269] 0929 14:11:49.963781 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:50.007684 (+ 43903us) service_pool.cc:227] Handling call
0929 14:11:52.502429 (+2494745us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:11:52.543018 (+ 40589us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6228864}
I20260929 14:11:52.544116 6845 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } attrs { replace: true } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.544839 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.544947 6879 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.526875 5347 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:11:52.545959 5347 raft_consensus.cc:595] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:52.546315 5347 raft_consensus.cc:603] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:11:52.520953 5198 rpcz_store.cc:276] 0929 14:11:50.054004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:50.058965 (+ 4961us) service_pool.cc:227] Handling call
0929 14:11:50.060133 (+ 1168us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:50.060333 (+ 200us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:50.060361 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:50.317102 (+256741us) spinlock_profiling.cc:243] Waited 1.81 ms on lock 0x7b2c0005ac28. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda3450e11 00007fbda3460442 00007fbda4aa8c11 00007fbda4a6d58a 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772
0929 14:11:50.317531 (+ 429us) log.cc:842] Serialized 26 byte log entry
0929 14:11:50.389811 (+ 72280us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:11:50.389851 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:50.389920 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:52.512503 (+2122583us) raft_consensus.cc:1671] finished
0929 14:11:52.512562 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:52.512872 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5896832,"thread_start_us":71891,"threads_started":1,"wal-append.queue_time_us":125348,"wal-append.run_cpu_time_us":2192,"wal-append.run_wall_time_us":1029587}
I20260929 14:11:52.532294 6662 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:11:52.543131 6834 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.535813 5349 raft_consensus.cc:2454] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 1.
I20260929 14:11:52.547730 6834 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.548072 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.543103 5164 leader_election.cc:401] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260929 14:11:52.548934 5347 raft_consensus.cc:2379] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:52.549700 6834 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.550508 6871 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.543015 6870 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.551098 6871 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.553586 6826 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.554339 6826 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.554811 6642 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.555646 6642 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.557070 6844 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.557909 6642 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.557780 6844 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.558189 6650 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.558584 6662 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.558892 6650 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.560083 6844 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.560684 6662 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.676364 6675 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:52.642222 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:52.575527 6662 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.664415 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.625520 6732 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.676616 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:52.627599 6550 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:52.594563 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.681619 6732 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.682149 6550 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.595185 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.682691 5902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:52.679798 6879 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.684263 6732 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.684510 5902 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:11:52.684826 6732 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.684983 6550 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.685622 6550 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.685493 6732 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:52.643038 5273 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50776 (request call id 1265) took 1921 ms (client timeout 1826 ms). Trace:
I20260929 14:11:52.740299 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:52.740674 5273 rpcz_store.cc:269] 0929 14:11:50.721523 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:52.642674 (+1921151us) service_pool.cc:214] Skipping call since client already timed out
0929 14:11:52.642846 (+ 172us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260929 14:11:52.741982 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.742758 5274 raft_consensus.cc:2379] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:52.563917 6585 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.123s
W20260929 14:11:52.638188 5271 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1619) took 2567 ms (client timeout 1529 ms). Trace:
I20260929 14:11:52.628262 6657 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.687060 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:52.613819 6635 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.627959 6770 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.646520 6619 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:52.682400 5268 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
W20260929 14:11:52.744103 5271 rpcz_store.cc:269] 0929 14:11:50.070529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:52.637835 (+2567306us) service_pool.cc:214] Skipping call since client already timed out
0929 14:11:52.638004 (+ 169us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260929 14:11:52.666098 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:52.680981 6662 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.589809 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.520114 6878 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.753970 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.745354 6657 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.755193 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.755986 6657 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.756273 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.745524 5904 raft_consensus.cc:2379] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 5: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
W20260929 14:11:52.644838 5192 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.306s user 0.023s sys 0.014s
I20260929 14:11:52.561080 6642 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.745713 6770 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.745846 6619 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.757711 6642 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.758584 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:52.759687 6770 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.670047 5096 catalog_manager.cc:5495] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 2 to 3, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 3 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:52.665635 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:52.759975 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:52.746872 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.760890 5274 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:52.754671 6878 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.759907 6642 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.761267 5096 catalog_manager.cc:5495] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 2 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:11:52.746138 5316 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:52.763659 6878 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.745347 6635 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.765519 5903 raft_consensus.cc:2454] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:52.645684 6686 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.748625 6662 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.766826 6686 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.767381 6635 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.759688 6619 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.746762 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:11:52.757501 5192 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.419s user 0.024s sys 0.015s
I20260929 14:11:52.768745 5268 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:52.768862 6686 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.592116 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.635685 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.690160 6879 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.561142 6650 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.598347 5272 raft_consensus.cc:1261] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260929 14:11:52.566597 5267 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50776 (request call id 1249) took 2348 ms (client timeout 1742 ms). Trace:
W20260929 14:11:52.691779 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 150214us
I20260929 14:11:52.630909 6659 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:52.691946 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:52.612004 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:52.680876 5907 raft_consensus.cc:2454] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:52.593726 6479 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:52.553139 5870 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:52.561375 6731 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.680433 6675 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.768141 6635 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:11:52.780790 6635 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:52.780638 5269 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:52.781160 6635 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:11:52.746901 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.782328 6675 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.782680 5271 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:52.565022 6870 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.594179 6834 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.763015 5096 catalog_manager.cc:5495] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 1 to 2, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 2 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:52.561655 6661 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.784591 6661 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.785116 5164 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:52.785189 5096 catalog_manager.cc:5495] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 2 to 3, leader changed from 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 3 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:52.788189 5196 raft_consensus.cc:2396] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260929 14:11:52.562009 6826 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.695832 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.673312 5349 raft_consensus.cc:1226] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:52.628306 6871 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.696650 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:52.525938 5195 rpcz_store.cc:269] 0929 14:11:50.097055 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:50.126532 (+ 29477us) service_pool.cc:227] Handling call
0929 14:11:52.495524 (+2368992us) spinlock_profiling.cc:243] Waited 23.7 ms on lock 0x7b54003db654. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:52.495675 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":54590336}
I20260929 14:11:52.698252 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:52.802902 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.687070 5345 raft_consensus.cc:2396] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260929 14:11:52.698822 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:52.701575 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:52.703590 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:52.705391 6674 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:52.618440 5130 debug-util.cc:398] Leaking SignalData structure 0x7b0800033ac0 after lost signal to thread 5133
I20260929 14:11:52.586035 6742 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:52.629949 6845 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.698949 6883 raft_consensus.cc:2726] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5.
I20260929 14:11:52.806353 5200 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:52.807250 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:52.711421 6844 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.807845 5199 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:52.808092 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:52.807994 6844 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.713899 6550 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.809095 5199 raft_consensus.cc:1226] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260929 14:11:52.810771 6550 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.810833 6844 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.675345 5099 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52736 (request call id 222) took 8109 ms. Trace:
I20260929 14:11:52.811434 6742 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.503019 6874 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.720661 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.514665 6872 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:52.721927 5165 connection.cc:232] RPC call timeout handler was delayed by -1.1909429629999977s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1529s for a response
I20260929 14:11:52.722836 6751 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.723841 6811 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.817965 6751 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.818117 6811 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.811709 5099 rpcz_store.cc:276] 0929 14:11:44.565873 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:44.576210 (+ 10337us) service_pool.cc:227] Handling call
0929 14:11:52.675123 (+8098913us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:11:48.612626 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:11:48.728679 (+116053us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:11:48.728714 (+ 35us) write_op.cc:435] Acquired schema lock
0929 14:11:48.728734 (+ 20us) tablet.cc:660] Decoding operations
0929 14:11:48.830075 (+101341us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:11:48.836939 (+ 6864us) write_op.cc:641] Partition lock acquired for write op
0929 14:11:48.841840 (+ 4901us) tablet.cc:683] Acquiring locks for 8 operations
0929 14:11:48.886514 (+ 44674us) tablet.cc:699] Row locks acquired
0929 14:11:48.889131 (+ 2617us) write_op.cc:260] PREPARE: finished
0929 14:11:48.900718 (+ 11587us) write_op.cc:270] Start()
0929 14:11:48.915215 (+ 14497us) write_op.cc:276] Timestamp: P: 1790691108898312 usec, L: 0
0929 14:11:48.915243 (+ 28us) op_driver.cc:331] REPLICATION: starting
0929 14:11:49.019483 (+104240us) log.cc:842] Serialized 3234 byte log entry
0929 14:11:49.934010 (+914527us) op_driver.cc:444] REPLICATION: finished
0929 14:11:50.167752 (+233742us) write_op.cc:301] APPLY: starting
0929 14:11:50.167848 (+ 96us) tablet.cc:1365] starting BulkCheckPresence
0929 14:11:50.169154 (+ 1306us) tablet.cc:1368] finished BulkCheckPresence
0929 14:11:50.206472 (+ 37318us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:11:50.260876 (+ 54404us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:11:50.261100 (+ 224us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0929 14:11:50.261145 (+ 45us) write_op.cc:312] APPLY: finished
0929 14:11:50.261506 (+ 361us) log.cc:842] Serialized 65 byte log entry
0929 14:11:50.261659 (+ 153us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:50.261799 (+ 140us) write_op.cc:454] Released schema lock
0929 14:11:50.261857 (+ 58us) write_op.cc:341] FINISH: Updating metrics
0929 14:11:50.262109 (+ 252us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:50.262140 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":224445,"apply.run_cpu_time_us":3971,"apply.run_wall_time_us":95810,"num_ops":8,"prepare.queue_time_us":115980,"prepare.run_cpu_time_us":50779,"prepare.run_wall_time_us":649173,"raft.queue_time_us":443255,"raft.run_cpu_time_us":14788,"raft.run_wall_time_us":240872,"replication_time_us":1008266,"spinlock_wait_cycles":1002880,"thread_start_us":736389,"threads_started":4,"wal-append.queue_time_us":296625,"wal-append.run_cpu_time_us":47019,"wal-append.run_wall_time_us":3124560}]]}
I20260929 14:11:52.731668 6736 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.820186 6751 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.732398 6717 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.721932 6885 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.734331 6476 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.822860 6493 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.824455 6659 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.825394 6871 raft_consensus.cc:2781] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:52.824874 6476 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.825775 6871 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.826191 6871 raft_consensus.cc:3037] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:52.826763 6659 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.827456 6476 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.734495 6718 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.828150 6872 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.828305 6718 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:52.829478 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 278400us
I20260929 14:11:52.830408 6872 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.830928 6718 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.734611 6629 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.816104 6874 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.831864 6629 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.834456 6629 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.737664 6707 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.835397 6811 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.769564 5192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1260) took 2504 ms. Trace:
I20260929 14:11:52.835172 6707 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.734726 6403 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.742803 5273 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:52.655012 5343 raft_consensus.cc:1226] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:52.842864 6733 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.843151 6707 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.842901 6403 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.843438 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.844122 5343 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:52.512061 6875 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.650856 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:52.549485 6654 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.847004 6654 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.783631 6834 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.589190 6587 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:52.847846 6587 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.848937 6834 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.849359 6717 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.851717 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.851967 6717 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.813050 6867 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:52.852952 6867 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.843464 6733 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.854357 5268 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:52.855284 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.852972 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.856694 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.859068 6733 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.858690 6871 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.786726 6642 raft_consensus.cc:2781] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:11:52.861261 6642 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:52.861743 6642 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:52.839874 6731 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.862402 6871 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.863183 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:52.863755 5907 raft_consensus.cc:2454] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:52.863839 6731 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.860288 5900 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45248 (request call id 239) took 2764 ms. Trace:
I20260929 14:11:52.864639 6479 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.791910 5351 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:52.865247 5900 rpcz_store.cc:276] 0929 14:11:50.095997 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:50.098446 (+ 2449us) service_pool.cc:227] Handling call
0929 14:11:52.511076 (+2412630us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b5800027698. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0dbce 000000000034aea9 00007fbda9939a30 00007fbda99ff7b4 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:11:52.511304 (+ 228us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:52.742306 (+231002us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:52.742344 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:52.742841 (+ 497us) log.cc:842] Serialized 26 byte log entry
0929 14:11:52.837797 (+ 94956us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:11:52.842387 (+ 4590us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:52.859823 (+ 17436us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:52.859877 (+ 54us) raft_consensus.cc:1671] finished
0929 14:11:52.859897 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:52.860114 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13989,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":160,"spinlock_wait_cycles":3011456,"thread_start_us":309834,"threads_started":2,"wal-append.queue_time_us":63498}
I20260929 14:11:52.866839 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.553299 6728 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.793254 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:52.835971 5192 rpcz_store.cc:276] 0929 14:11:50.264972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:50.308937 (+ 43965us) service_pool.cc:227] Handling call
0929 14:11:50.337006 (+ 28069us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:50.337849 (+ 843us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:50.337932 (+ 83us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:50.338430 (+ 498us) log.cc:842] Serialized 26 byte log entry
0929 14:11:52.768976 (+2430546us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:52.769055 (+ 79us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:52.769174 (+ 119us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:52.769260 (+ 86us) raft_consensus.cc:1671] finished
0929 14:11:52.769277 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:52.769493 (+ 216us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1684,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":15413,"thread_start_us":2430041,"threads_started":1,"wal-append.queue_time_us":126588,"wal-append.run_cpu_time_us":2184,"wal-append.run_wall_time_us":1005441}
I20260929 14:11:52.801980 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.875932 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.876492 5269 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:52.876942 5272 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:52.806320 5194 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
W20260929 14:11:52.812680 5165 connection.cc:232] RPC call timeout handler was delayed by -1.0801179979998778s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15680000000000002s for a response
I20260929 14:11:52.801131 5096 catalog_manager.cc:5495] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 1 to 2, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 2 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:52.873920 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.814453 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.816258 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.816923 6742 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.817061 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.724987 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.731845 6583 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:52.820698 5344 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:52.881717 5197 raft_consensus.cc:1226] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:52.827855 5347 raft_consensus.cc:2379] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:52.830276 5870 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:52.737003 5314 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:52.738366 6868 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:52.884199 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.884346 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.884655 6868 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.885413 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:52.886276 5274 raft_consensus.cc:2354] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:11:52.887125 5276 raft_consensus.cc:2396] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260929 14:11:52.748345 6732 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.834460 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.819993 6736 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.824899 6493 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.786624 6661 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.860742 6874 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.747258 5346 raft_consensus.cc:2379] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:52.891194 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.873759 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.874248 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:52.906147 5907 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:11:52.874774 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:52.844995 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.875075 5195 raft_consensus.cc:1226] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:52.875826 5198 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
W20260929 14:11:52.789191 5267 rpcz_store.cc:269] 0929 14:11:50.217708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:52.566146 (+2348438us) service_pool.cc:214] Skipping call since client already timed out
0929 14:11:52.566338 (+ 192us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260929 14:11:52.809091 5200 raft_consensus.cc:2354] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6.
I20260929 14:11:52.907545 5276 raft_consensus.cc:2354] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:11:52.878676 5271 raft_consensus.cc:2454] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:52.877445 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.769724 6741 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:52.919001 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.919219 5242 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:52.919521 6629 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:11:52.919397 6741 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.920166 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.920258 6629 raft_consensus.cc:686] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:52.920531 5242 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:52.921494 6629 consensus_queue.cc:237] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.923758 6493 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.849987 6587 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.921562 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.925051 5269 raft_consensus.cc:2454] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
W20260929 14:11:52.925290 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 95108.3us
I20260929 14:11:52.925127 6741 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.893537 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:52.926091 5346 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 6
I20260929 14:11:52.926070 5870 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:52.921487 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:52.927053 6728 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:52.928439 5242 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:52.929036 5269 raft_consensus.cc:1226] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260929 14:11:52.930173 5269 raft_consensus.cc:1226] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:52.930441 6742 raft_consensus.cc:2726] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:52.930712 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.931098 6742 raft_consensus.cc:2726] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:52.932164 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:52.824251 6885 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.933523 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:52.934458 6885 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:52.935236 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.935984 5351 raft_consensus.cc:2379] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 2: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:52.936045 5239 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:52.936987 6619 consensus_queue.cc:1046] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:11:52.892759 6583 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:52.938083 6619 consensus_queue.cc:1046] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:52.939185 6583 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.939309 6619 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:52.940269 6685 raft_consensus.cc:3032] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:52.940686 6685 raft_consensus.cc:729] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:52.941771 6685 consensus_queue.cc:260] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:52.942693 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 176256us
I20260929 14:11:52.943872 6686 raft_consensus.cc:3032] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:52.943670 6642 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.944339 6686 raft_consensus.cc:729] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
W20260929 14:11:52.893576 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 156846us
I20260929 14:11:52.886982 6868 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.890118 5349 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:52.880568 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:52.918210 5906 raft_consensus.cc:2454] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:52.918704 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.883895 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.949230 5164 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:52.893936 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.925555 6587 raft_consensus.cc:2726] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:52.935684 5242 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:52.921149 5344 raft_consensus.cc:2396] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260929 14:11:52.910900 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:52.919435 5267 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:48196
I20260929 14:11:52.952438 5342 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:52.803752 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:52.945171 5316 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:52.969696 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.972914 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.973960 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.974490 5349 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:52.975069 5316 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:52.975258 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.976284 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.945441 6686 consensus_queue.cc:260] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:52.976841 5349 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:52.946851 5275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1264) took 2100 ms. Trace:
I20260929 14:11:52.905122 6479 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.977779 6686 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:52.978176 5871 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:52.978332 6822 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:52.978536 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.978897 6735 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:52.978209 5242 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:52.979754 6736 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:52.981632 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.982154 5164 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:52.888901 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:52.904954 6403 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:52.946550 5350 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:11:53.009009 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:52.984217 6885 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:52.985435 5275 rpcz_store.cc:276] 0929 14:11:50.721175 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:52.487002 (+1765827us) service_pool.cc:227] Handling call
0929 14:11:52.487776 (+ 774us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:52.488403 (+ 627us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:52.488443 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:52.489271 (+ 828us) log.cc:842] Serialized 26 byte log entry
0929 14:11:52.804449 (+315178us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:52.804493 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:52.804557 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:52.820901 (+ 16344us) raft_consensus.cc:1671] finished
0929 14:11:52.820921 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:52.821047 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":8548,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":214,"thread_start_us":314777,"threads_started":1,"wal-append.queue_time_us":315152}
I20260929 14:11:52.950846 5345 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:11:52.992055 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:52.998361 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:53.009652 5198 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
W20260929 14:11:52.984787 5130 debug-util.cc:398] Leaking SignalData structure 0x7b080030cfa0 after lost signal to thread 5225
I20260929 14:11:52.948755 6642 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:53.018684 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:53.019042 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.019686 5194 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:52.999379 6822 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:53.020289 6822 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:53.020597 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.021474 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:53.022054 5903 raft_consensus.cc:2454] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:52.948127 6751 consensus_queue.cc:1046] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.060s
I20260929 14:11:53.024526 6751 raft_consensus.cc:3032] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:53.024870 6751 raft_consensus.cc:729] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:53.025739 6751 consensus_queue.cc:260] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:53.026907 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.027697 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.000454 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:53.013743 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:52.945510 6619 raft_consensus.cc:686] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:52.984319 5348 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:52.980917 6871 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:53.014640 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:53.013666 5314 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Timed out: Call waited in the queue past client deadline [suppressed 5 similar messages]
I20260929 14:11:53.055120 5903 raft_consensus.cc:1261] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:53.046208 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.008404 5351 raft_consensus.cc:2396] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260929 14:11:52.946758 6583 consensus_queue.cc:1046] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
I20260929 14:11:53.047669 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:53.043889 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 65805.3us
I20260929 14:11:53.063285 5350 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:53.043042 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:53.064216 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:52.971349 5267 raft_consensus.cc:595] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:53.064649 5242 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:53.008574 6732 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260929 14:11:53.048328 6562 consensus_queue.cc:1159] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ac17d3b553b74c9da39e0ab307addacd" responder_term: 2 status { last_received { term: 2 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 2 index: 2 } }
I20260929 14:11:53.051748 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:53.018126 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.042291 6619 consensus_queue.cc:237] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:53.001267 6735 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:52.943638 6685 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:53.067099 6735 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:53.046679 6751 raft_consensus.cc:3037] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:53.012917 5349 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:53.028545 5347 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 1.
I20260929 14:11:53.064909 5351 raft_consensus.cc:3037] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:53.070613 5346 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
W20260929 14:11:53.013854 5130 debug-util.cc:398] Leaking SignalData structure 0x7b0800384000 after lost signal to thread 5226
I20260929 14:11:53.037639 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:53.073048 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:53.038749 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:53.041874 5240 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.3.242.126:41327, user_credentials={real_user=slave}} blocked reactor thread for 46577.4us
I20260929 14:11:53.007107 5871 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:53.002978 5316 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:53.064309 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:53.010933 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.017295 5274 raft_consensus.cc:2454] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:53.064776 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:53.065188 5267 raft_consensus.cc:603] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:11:53.061345 6871 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:53.018040 5276 raft_consensus.cc:1261] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260929 14:11:53.048095 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:53.014946 5270 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:53.037731 5197 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:53.012661 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:53.101163 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:53.074124 5164 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:53.009325 6867 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:53.072649 5239 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:53.103389 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:53.011912 6885 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:53.044214 6742 raft_consensus.cc:2781] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:53.092839 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:11:53.095304 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 88269.6us
I20260929 14:11:53.049676 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:53.011889 6654 raft_consensus.cc:2781] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:53.056318 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:53.029284 5194 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:53.049481 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:53.051198 6822 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:53.041749 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.176213 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:53.044900 5902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:11:53.108791 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 105946us
I20260929 14:11:52.948216 5268 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:53.065490 6732 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:53.043202 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:53.103307 6871 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:53.105443 5348 raft_consensus.cc:2454] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:53.143097 6868 raft_consensus.cc:3032] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:53.104177 6874 raft_consensus.cc:2781] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260929 14:11:53.144037 5130 debug-util.cc:398] Leaking SignalData structure 0x7b080022db20 after lost signal to thread 5300
I20260929 14:11:53.107988 6885 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:53.066913 5196 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:53.146204 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:53.147320 5164 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:53.042752 6886 log.cc:924] Time spent T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260929 14:11:53.030937 5870 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:53.146090 6686 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:53.138128 6642 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:11:53.181152 6822 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:53.177829 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:53.217496 5316 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:53.178237 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 113708us
I20260929 14:11:53.177459 5871 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:53.218679 5239 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:53.152287 5349 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:11:53.217815 5903 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 1.
I20260929 14:11:53.211026 5351 raft_consensus.cc:2454] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:53.223459 5351 raft_consensus.cc:1261] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260929 14:11:53.223841 5351 pending_rounds.cc:76] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4: Aborting all ops after (but not including) 1
I20260929 14:11:53.225322 5347 raft_consensus.cc:1226] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
W20260929 14:11:53.225423 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 48041.3us
I20260929 14:11:53.152776 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:11:53.226140 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 165860us
I20260929 14:11:53.226631 5907 raft_consensus.cc:2379] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 2: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:11:53.153412 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:53.224769 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 77490.3us
I20260929 14:11:53.108085 6742 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:53.187345 5167 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:53.228535 5164 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:53.228972 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 78666.3us
I20260929 14:11:53.230012 5239 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:53.224093 5351 pending_rounds.cc:97] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260929 14:11:53.231537 5351 raft_consensus.cc:2866] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:11:53.227365 5350 raft_consensus.cc:2454] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 1.
I20260929 14:11:53.177184 5905 raft_consensus.cc:2379] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 5: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:11:53.182963 5192 raft_consensus.cc:2396] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for term 5 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260929 14:11:53.138447 6867 raft_consensus.cc:2781] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:53.248401 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:53.136922 6654 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:53.233963 5164 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:53.151427 6728 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260929 14:11:53.247754 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 60649.1us
I20260929 14:11:53.198436 5901 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:11:53.182994 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:53.247658 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:53.224798 6874 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:53.225363 6868 raft_consensus.cc:729] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:53.150665 5273 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:53.217633 6642 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 14:11:53.251441 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 237801us
I20260929 14:11:53.254390 5167 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:53.244308 6879 raft_consensus.cc:2781] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:53.253590 5907 raft_consensus.cc:2454] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:11:53.224897 6732 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:53.255885 5906 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:53.251772 6741 raft_consensus.cc:2781] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:53.197885 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:53.176906 5343 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:11:53.234140 6742 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:53.257092 5316 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:53.233922 6583 raft_consensus.cc:2741] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260929 14:11:53.248049 6675 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:11:53.291880 6675 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:53.292277 6675 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:53.256568 6479 consensus_queue.cc:1035] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260929 14:11:53.293833 6868 consensus_queue.cc:260] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:53.295153 6868 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:53.285641 6741 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:53.297680 6741 raft_consensus.cc:3037] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
W20260929 14:11:53.297758 6892 log.cc:924] Time spent T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.134s user 0.000s sys 0.001s
I20260929 14:11:53.249316 6872 raft_consensus.cc:2781] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:53.246200 6731 raft_consensus.cc:2781] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:53.266165 5164 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:53.216810 6686 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:11:53.246577 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 234883us
I20260929 14:11:53.312243 5870 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:53.250443 6867 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:53.313256 6867 raft_consensus.cc:3037] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:53.317724 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.252234 6845 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:53.253155 6654 raft_consensus.cc:3037] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:53.320281 5164 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:53.300539 6707 raft_consensus.cc:2781] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:53.321672 6707 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:53.321843 6550 raft_consensus.cc:2781] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:11:53.322042 6707 raft_consensus.cc:3037] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:53.275386 5314 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
W20260929 14:11:53.323935 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 48634.9us
I20260929 14:11:53.284168 5167 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:53.257829 6874 raft_consensus.cc:3037] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:53.282249 6659 raft_consensus.cc:2781] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:53.288702 5316 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:53.256433 6661 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:11:53.306146 6674 raft_consensus.cc:2781] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:53.337491 6659 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:53.337597 6674 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:53.337972 6659 raft_consensus.cc:3037] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:53.338192 6674 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:53.287097 6844 consensus_queue.cc:1035] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.054s
I20260929 14:11:53.339609 6735 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:53.341405 6751 raft_consensus.cc:2781] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:53.341795 6751 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:53.341883 6735 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:53.342178 6751 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:53.342499 6728 raft_consensus.cc:2781] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:53.342927 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.343858 6562 raft_consensus.cc:2781] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260929 14:11:53.344060 6635 consensus_meta.cc:287] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.562s user 0.010s sys 0.000s
I20260929 14:11:53.344333 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.345081 5198 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
W20260929 14:11:53.299274 5130 debug-util.cc:398] Leaking SignalData structure 0x7b0800249f40 after lost signal to thread 5375
W20260929 14:11:53.346035 5130 debug-util.cc:398] Leaking SignalData structure 0x7b08003283e0 after lost signal to thread 5376
I20260929 14:11:53.318897 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.346912 5193 raft_consensus.cc:2379] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 2: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:53.343006 6728 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:53.347582 6728 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:53.344378 6562 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:53.345458 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:53.333746 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 45143.7us
I20260929 14:11:53.351301 5270 raft_consensus.cc:2454] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:53.367472 5314 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:53.278925 6642 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:11:53.383260 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 100083us
I20260929 14:11:53.383952 5314 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:53.385154 5314 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:53.387058 5314 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:53.301276 6872 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:53.391460 5242 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:53.302615 6731 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:53.330406 6686 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:53.394788 5314 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:53.401041 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:53.401865 5197 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:53.364928 6562 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:53.344513 6635 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:53.405817 6635 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:53.408268 5346 raft_consensus.cc:2379] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 6: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:53.382860 6834 raft_consensus.cc:2726] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:53.383771 6661 raft_consensus.cc:686] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:53.391399 6717 raft_consensus.cc:2781] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260929 14:11:53.423626 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 100128us
I20260929 14:11:53.399864 6731 raft_consensus.cc:3037] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
W20260929 14:11:53.390205 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 78068.6us
I20260929 14:11:53.397014 6872 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:53.396250 6822 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:53.428203 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.392234 6718 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:53.440167 5275 raft_consensus.cc:1226] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:53.399389 6811 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260929 14:11:53.454155 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 177019us
I20260929 14:11:53.438740 5316 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:53.456998 5316 leader_election.cc:398] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6.
I20260929 14:11:53.399609 6879 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:53.411473 6823 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:53.457583 5316 leader_election.cc:401] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
W20260929 14:11:53.446018 5130 debug-util.cc:398] Leaking SignalData structure 0x7b08003c1040 after lost signal to thread 5932
I20260929 14:11:53.399258 6476 consensus_queue.cc:1035] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.047s
I20260929 14:11:53.445812 6550 raft_consensus.cc:686] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:53.441588 6717 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:53.449717 6718 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:53.449503 6822 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:53.459183 6811 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:11:53.465987 5130 debug-util.cc:398] Leaking SignalData structure 0x7b08003c1060 after lost signal to thread 5933
I20260929 14:11:53.463495 6823 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:53.458633 6629 consensus_queue.cc:1046] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
I20260929 14:11:53.479648 6583 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6.
I20260929 14:11:53.465050 6493 raft_consensus.cc:2781] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260929 14:11:53.496026 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 211948us
I20260929 14:11:53.476920 6717 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:53.497896 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.461531 6879 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:53.508772 6823 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:53.485646 6718 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:53.485352 6822 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:53.542678 5167 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:53.488837 6811 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:11:53.548202 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 81364.8us
I20260929 14:11:53.549122 6845 raft_consensus.cc:686] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:53.551981 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.552625 5198 raft_consensus.cc:2454] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 1.
I20260929 14:11:53.553984 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.526685 6493 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:53.551108 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:53.561503 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:53.562237 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:53.546465 6403 raft_consensus.cc:3032] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Stepping down as leader of term 2
W20260929 14:11:53.556628 5130 debug-util.cc:398] Leaking SignalData structure 0x7b08002a5700 after lost signal to thread 6744
I20260929 14:11:53.563544 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:53.596856 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 98065.2us
I20260929 14:11:53.597141 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:53.597859 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:53.598344 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:53.599076 5275 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:53.599366 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:53.596642 6845 consensus_queue.cc:237] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:53.599946 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:53.598470 5276 raft_consensus.cc:2454] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:53.544842 6906 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:53.602912 5873 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:53.604779 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 220877us
I20260929 14:11:53.560753 6493 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:53.617910 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:53.602835 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 126869us
I20260929 14:11:53.546727 6736 consensus_queue.cc:1046] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
I20260929 14:11:53.573405 5192 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 3
W20260929 14:11:53.625327 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 170412us
I20260929 14:11:53.521632 6661 consensus_queue.cc:237] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:53.548071 6742 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:53.573654 6403 raft_consensus.cc:729] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:53.496017 5870 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:53.560976 6629 consensus_queue.cc:1035] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20260929 14:11:53.601640 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:53.629364 6619 raft_consensus.cc:2781] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:53.649401 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:53.604257 6654 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:53.651976 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 109382us
W20260929 14:11:53.656800 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 64547.8us
I20260929 14:11:53.693535 5167 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:53.702448 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:53.702994 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 46496.9us
I20260929 14:11:53.704468 5276 raft_consensus.cc:2454] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:53.706619 5903 raft_consensus.cc:1226] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:53.704334 5167 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:53.601183 6550 consensus_queue.cc:237] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:53.647089 6619 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:53.708051 5239 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:53.705519 6479 consensus_queue.cc:1046] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260929 14:11:53.656201 6913 raft_consensus.cc:3037] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:53.754364 6902 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:53.754995 6902 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:53.712054 6867 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:53.654865 6675 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:53.767721 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 148078us
I20260929 14:11:53.758580 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:53.706442 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:53.845554 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:53.846379 5276 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:53.843673 5316 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:53.850790 5316 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:53.750154 6619 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:53.749271 6906 raft_consensus.cc:2781] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:53.751827 6870 raft_consensus.cc:2781] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:53.873056 5199 raft_consensus.cc:1226] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 4. Current term is 5. Ops: []
W20260929 14:11:53.883492 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 181246us
I20260929 14:11:53.886015 5344 raft_consensus.cc:1226] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:53.895183 6834 raft_consensus.cc:2781] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:53.844246 6629 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:53.887784 6685 consensus_meta.cc:287] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.821s user 0.012s sys 0.005s
W20260929 14:11:53.925887 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 323030us
I20260929 14:11:53.930286 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:53.933156 5267 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:53.885681 6906 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:53.931068 5873 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:53.926584 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 74827.2us
I20260929 14:11:53.887125 5268 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
W20260929 14:11:53.926546 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 222272us
I20260929 14:11:53.949877 5316 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:53.951303 5167 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:53.932945 6915 raft_consensus.cc:2781] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:53.925077 6908 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:53.928189 6834 raft_consensus.cc:686] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:53.916797 6902 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:53.925345 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:53.822546 6403 consensus_queue.cc:260] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:53.935340 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:53.964936 5316 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:53.932196 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 447001us
I20260929 14:11:53.964763 6915 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:53.937917 6905 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:53.949198 6906 raft_consensus.cc:3037] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:53.988824 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:11:53.965050 5164 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.011s sys 0.014s
W20260929 14:11:54.032456 5164 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.584s user 0.015s sys 0.018s
I20260929 14:11:53.937731 6916 raft_consensus.cc:2726] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260929 14:11:54.026618 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 61806.4us
I20260929 14:11:54.070398 5316 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:54.070560 6675 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:54.071357 5316 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:54.071544 6867 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:53.983431 6751 consensus_meta.cc:287] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.641s user 0.034s sys 0.021s
I20260929 14:11:54.072481 6867 raft_consensus.cc:2781] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:11:54.072343 6751 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.072842 6867 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:53.959314 6629 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.073179 6867 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:54.074666 5242 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:54.028947 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.076339 6902 raft_consensus.cc:2726] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:53.974568 6742 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.031922 6476 raft_consensus.cc:2781] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260929 14:11:54.076363 5901 consensus_meta.cc:287] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.791s user 0.008s sys 0.004s
I20260929 14:11:54.077566 6476 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:54.077798 5901 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:11:53.927120 6635 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:11:54.078584 6635 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:54.079149 6635 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:54.079310 6924 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:11:54.042309 5272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:50768 (request call id 893) took 1298 ms. Trace:
I20260929 14:11:53.937430 6921 consensus_queue.cc:1046] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.110s
I20260929 14:11:54.081629 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.082304 6921 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:54.047586 6917 raft_consensus.cc:2726] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:54.051227 6928 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:54.081346 5272 rpcz_store.cc:276] 0929 14:11:52.743647 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:53.046004 (+302357us) service_pool.cc:227] Handling call
0929 14:11:54.041888 (+995884us) spinlock_profiling.cc:243] Waited 889 ms on lock 0x7b54003741d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2af0 00007fbda4a75f2a 00007fbda99b2c6e 00007fbda30a9807 00007fbda30a9792 00007fbda30a96c1 00007fbda30a963c 00007fbda30a88f2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:54.042226 (+ 338us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2045479680}
I20260929 14:11:54.083033 6917 raft_consensus.cc:3032] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:54.083376 6917 raft_consensus.cc:729] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
W20260929 14:11:54.025416 6875 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.558s user 0.017s sys 0.017s
W20260929 14:11:54.083881 6875 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.621s user 0.018s sys 0.018s
I20260929 14:11:54.084486 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.084252 6917 consensus_queue.cc:260] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.056438 6915 raft_consensus.cc:3037] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:54.031702 6908 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.027294 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:54.058737 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:54.061371 5906 raft_consensus.cc:1261] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260929 14:11:54.064803 6910 log.cc:924] Time spent T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.351s user 0.004s sys 0.007s
I20260929 14:11:54.065587 6914 raft_consensus.cc:3032] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:54.055439 6654 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:54.065657 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:54.065898 5344 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 1727) took 1119 ms. Trace:
I20260929 14:11:54.067456 5903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45248 (request call id 262) took 1016 ms. Trace:
I20260929 14:11:54.068153 6834 consensus_queue.cc:237] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.068845 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:54.071436 6909 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:54.077978 6476 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:54.078526 5901 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45232 (request call id 294) took 1030 ms. Trace:
I20260929 14:11:54.038969 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:54.051838 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:54.057592 6885 consensus_meta.cc:287] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.810s user 0.015s sys 0.040s
I20260929 14:11:54.098379 5199 raft_consensus.cc:2379] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 3: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:54.087811 5316 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:54.128463 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:54.032881 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 590484us
W20260929 14:11:54.103786 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 218987us
I20260929 14:11:54.085099 5272 raft_consensus.cc:2362] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5. Re-sending same reply.
W20260929 14:11:54.134426 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 205099us
W20260929 14:11:54.138656 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 66519us
I20260929 14:11:54.104180 6870 raft_consensus.cc:686] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:54.112056 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.108160 5344 rpcz_store.cc:276] 0929 14:11:52.946687 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:53.224946 (+278259us) service_pool.cc:227] Handling call
0929 14:11:53.251963 (+ 27017us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:53.886527 (+634564us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:54.065605 (+179078us) spinlock_profiling.cc:243] Waited 611 ms on lock 0x7b54003a9dd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:54.065803 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1404676992}
I20260929 14:11:54.109020 5903 rpcz_store.cc:276] 0929 14:11:53.050868 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:53.256544 (+205676us) service_pool.cc:227] Handling call
0929 14:11:53.282332 (+ 25788us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:53.706981 (+424649us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:54.067218 (+360237us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b540060f9d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:54.067385 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":975361408}
W20260929 14:11:54.095367 5197 consensus_meta.cc:287] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.672s user 0.012s sys 0.016s
I20260929 14:11:54.142773 5197 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:54.138047 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.101027 6403 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:54.082633 6921 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:54.144063 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.074712 6629 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:54.146073 5198 raft_consensus.cc:2396] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260929 14:11:54.148363 5267 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:54.153826 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:54.155946 5271 raft_consensus.cc:1261] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:11:54.156432 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:54.156836 5271 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:48196
I20260929 14:11:54.157473 6900 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:54.157789 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:54.103008 6914 raft_consensus.cc:729] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:54.158948 5276 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:54.156970 6926 raft_consensus.cc:2726] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:11:54.102195 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 55442.7us
I20260929 14:11:54.159960 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.161413 5347 raft_consensus.cc:1226] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 5. Current term is 6. Ops: []
W20260929 14:11:54.162216 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 231229us
I20260929 14:11:54.163720 5316 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:54.171751 5873 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:54.168905 5342 raft_consensus.cc:1226] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:54.168818 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:54.140829 6917 raft_consensus.cc:3037] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:54.170257 5272 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:54.177716 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:54.158049 6900 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.184075 5873 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:54.152383 6921 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:54.186605 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.187697 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:54.189642 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.201455 5275 raft_consensus.cc:3037] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:54.188652 5906 raft_consensus.cc:3037] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:54.202235 5873 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:54.176843 6741 consensus_meta.cc:287] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.879s user 0.008s sys 0.019s
I20260929 14:11:54.203701 5267 raft_consensus.cc:2379] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 2: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:54.204062 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.205356 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:54.207353 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.128441 5901 rpcz_store.cc:276] 0929 14:11:53.048558 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:53.049023 (+ 465us) service_pool.cc:227] Handling call
0929 14:11:54.078427 (+1029404us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:54.167169 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.214624 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:54.204661 5272 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:54.208912 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
W20260929 14:11:54.189071 6686 consensus_meta.cc:287] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.776s user 0.019s sys 0.012s
W20260929 14:11:54.188604 6659 consensus_meta.cc:287] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.850s user 0.016s sys 0.016s
W20260929 14:11:54.186650 6674 consensus_meta.cc:287] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.848s user 0.014s sys 0.025s
I20260929 14:11:54.147882 6885 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.184008 6736 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
W20260929 14:11:54.210955 6879 consensus_meta.cc:287] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.659s user 0.013s sys 0.008s
I20260929 14:11:54.187608 6654 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260929 14:11:54.212031 6874 consensus_meta.cc:287] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.866s user 0.016s sys 0.006s
I20260929 14:11:54.200385 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:54.240871 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 54649.5us
I20260929 14:11:54.272205 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:11:54.218209 6717 consensus_meta.cc:287] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.672s user 0.005s sys 0.004s
W20260929 14:11:54.218358 6822 consensus_meta.cc:287] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.663s user 0.004s sys 0.005s
I20260929 14:11:54.276635 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.285172 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.287171 5276 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
W20260929 14:11:54.275290 6728 consensus_meta.cc:287] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.927s user 0.017s sys 0.003s
I20260929 14:11:54.286736 5270 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:54.239267 6931 raft_consensus.cc:2781] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260929 14:11:54.238613 6732 consensus_meta.cc:287] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.936s user 0.013s sys 0.001s
I20260929 14:11:54.281644 5873 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:54.282999 5197 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
W20260929 14:11:54.242130 6642 consensus_meta.cc:287] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.840s user 0.030s sys 0.071s
W20260929 14:11:54.239914 6871 consensus_meta.cc:287] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 1.006s user 0.008s sys 0.008s
W20260929 14:11:54.242519 6707 consensus_meta.cc:287] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.920s user 0.009s sys 0.005s
I20260929 14:11:54.241112 5904 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:11:54.277738 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.181041 6909 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.289204 5200 raft_consensus.cc:2379] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 4: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:54.215131 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:54.164419 6928 consensus_queue.cc:1035] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.115s
I20260929 14:11:54.320757 5316 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:54.217406 6741 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:54.271853 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 53689.3us
I20260929 14:11:54.286926 5268 raft_consensus.cc:2379] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 4: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:54.217875 6911 consensus_queue.cc:1046] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.085s
I20260929 14:11:54.365307 5870 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:54.365679 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.366353 5268 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:54.367182 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.367488 5906 raft_consensus.cc:2454] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:54.368705 5316 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:54.240221 6913 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:54.336097 5276 raft_consensus.cc:1261] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:11:54.282573 6736 raft_consensus.cc:686] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:54.370225 6812 raft_consensus.cc:2781] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:54.370769 6812 raft_consensus.cc:686] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
W20260929 14:11:54.371251 6872 consensus_meta.cc:287] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.903s user 0.007s sys 0.031s
I20260929 14:11:54.370821 6736 consensus_queue.cc:237] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.371625 6872 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.371743 6812 consensus_queue.cc:237] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.218683 6629 consensus_queue.cc:1046] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260929 14:11:54.373461 6935 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:54.373852 6629 raft_consensus.cc:3032] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:54.374249 6629 raft_consensus.cc:729] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:54.374409 5194 raft_consensus.cc:2379] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 2: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:54.294889 6931 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:54.375125 6629 consensus_queue.cc:260] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.337186 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:54.287251 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:54.345044 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.242497 6686 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.272166 6674 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.340832 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:54.349766 6868 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:54.316066 6728 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.272709 6659 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.288259 6717 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.353398 6583 raft_consensus.cc:2781] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:11:54.290175 6822 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.342518 6871 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.241976 6914 consensus_queue.cc:260] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.376120 6872 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.380705 6914 raft_consensus.cc:3037] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:54.381139 6872 raft_consensus.cc:2726] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:11:54.381970 5351 raft_consensus.cc:1226] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:54.339102 6642 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.363518 6930 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:54.364360 6741 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:54.344655 6927 log.cc:924] Time spent T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.128s user 0.001s sys 0.000s
I20260929 14:11:54.342574 6707 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.284689 6874 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.375649 6931 raft_consensus.cc:3037] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:54.398845 5201 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
W20260929 14:11:54.355264 5192 consensus_meta.cc:287] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.712s user 0.001s sys 0.006s
I20260929 14:11:54.408481 5192 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:54.331544 6751 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:54.409557 5196 raft_consensus.cc:2379] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 4: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:54.395000 6822 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:54.411983 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37270 (request call id 136) took 1098 ms. Trace:
I20260929 14:11:54.412353 5196 rpcz_store.cc:276] 0929 14:11:53.313092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:53.342585 (+ 29493us) service_pool.cc:227] Handling call
0929 14:11:54.411759 (+1069174us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54001d92d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:54.411903 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2429230848}
I20260929 14:11:54.415433 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:54.416941 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:54.280354 6879 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.393500 5903 raft_consensus.cc:1226] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:54.417550 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:54.417575 5199 raft_consensus.cc:2354] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:11:54.364108 6923 consensus_queue.cc:1046] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.079s
I20260929 14:11:54.335626 6907 consensus_queue.cc:1046] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
W20260929 14:11:54.380028 6731 consensus_meta.cc:287] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.921s user 0.015s sys 0.004s
I20260929 14:11:54.398756 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:54.433524 5197 raft_consensus.cc:1799] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Handling vote request from an unknown peer c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:54.282034 6870 consensus_queue.cc:237] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.398772 5270 raft_consensus.cc:1226] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
W20260929 14:11:54.398042 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 114408us
I20260929 14:11:54.402034 6885 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.402199 6728 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.402721 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:54.404222 6717 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:54.383102 6583 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:54.405368 5902 raft_consensus.cc:1226] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:54.405665 6871 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.397082 6930 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.368292 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.423278 6908 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.456971 5268 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:54.457252 5200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52182 (request call id 1295) took 1468 ms. Trace:
I20260929 14:11:54.398046 6476 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.457671 5200 rpcz_store.cc:276] 0929 14:11:52.988386 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:52.994679 (+ 6293us) service_pool.cc:227] Handling call
0929 14:11:54.456998 (+1462319us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54001450d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:54.457168 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2743085824}
I20260929 14:11:54.458319 6930 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.458762 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:54.459044 6476 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:54.459438 5901 raft_consensus.cc:1226] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260929 14:11:54.459906 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:54.460676 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:54.434042 6906 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.462241 6659 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:54.463256 6906 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:54.463917 5343 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260929 14:11:54.465425 5268 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:54.407557 6707 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:54.408032 5899 raft_consensus.cc:1226] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:54.408998 5192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52182 (request call id 1308) took 1010 ms. Trace:
W20260929 14:11:54.391865 6562 consensus_meta.cc:287] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.951s user 0.019s sys 0.022s
I20260929 14:11:54.282545 6916 consensus_queue.cc:1046] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.065s
I20260929 14:11:54.419961 5275 raft_consensus.cc:2454] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:11:54.422420 5906 raft_consensus.cc:2379] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 3: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:11:54.472721 5316 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:54.335019 6732 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.434051 5197 raft_consensus.cc:2379] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 2: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:54.384436 5314 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:54.435809 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:54.438267 5873 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:54.412616 6635 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.439652 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.434564 5196 raft_consensus.cc:2379] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 4: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:54.476212 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:54.476788 5197 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:54.477026 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:54.477878 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:54.478235 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:54.478899 5196 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:54.478999 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:54.480086 5196 raft_consensus.cc:1261] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:54.448374 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:54.450176 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:54.441566 6583 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:54.422883 5193 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
W20260929 14:11:54.498052 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 59874.7us
W20260929 14:11:54.454099 6493 consensus_meta.cc:287] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.826s user 0.011s sys 0.017s
W20260929 14:11:54.499523 6619 consensus_meta.cc:287] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.584s user 0.008s sys 0.000s
I20260929 14:11:54.455726 6674 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.499840 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.499780 6493 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.501327 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.501701 6900 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.502311 5268 raft_consensus.cc:2379] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:54.502589 6493 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:54.502748 6686 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.503225 5349 raft_consensus.cc:1226] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:54.503185 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.392731 6868 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.483554 5905 raft_consensus.cc:1226] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:54.508092 5167 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:54.509502 6868 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:54.508737 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.509836 6913 raft_consensus.cc:3032] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:11:54.510278 6913 raft_consensus.cc:729] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:54.510440 5195 raft_consensus.cc:2379] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 3: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:54.510550 5902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:54.511480 5195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52182 (request call id 1294) took 1523 ms. Trace:
I20260929 14:11:54.511272 6913 consensus_queue.cc:260] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.466825 5192 rpcz_store.cc:276] 0929 14:11:53.398480 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:53.398721 (+ 241us) service_pool.cc:227] Handling call
0929 14:11:53.550980 (+152259us) spinlock_profiling.cc:243] Waited 6.11 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0dbce 000000000034aea9 00007fbda9939a30 00007fbda99ff7b4 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:11:54.408941 (+857961us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14054272}
I20260929 14:11:54.511996 5268 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:54.512327 6885 consensus_queue.cc:1046] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:11:54.512796 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:54.513589 5192 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:54.514096 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.514566 6583 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.515228 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:54.515666 6933 log.cc:924] Time spent T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.236s user 0.001s sys 0.000s
I20260929 14:11:54.517846 5316 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:54.478397 6642 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.448169 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:54.481706 5201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52182 (request call id 1303) took 1085 ms. Trace:
I20260929 14:11:54.482323 5198 raft_consensus.cc:1226] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:54.518749 5201 rpcz_store.cc:276] 0929 14:11:53.396526 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:53.396676 (+ 150us) service_pool.cc:227] Handling call
0929 14:11:54.481553 (+1084877us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b5400012cd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:54.481652 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2187200768}
I20260929 14:11:54.486721 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:54.491861 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 107501us
I20260929 14:11:54.490860 5197 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:11:54.523487 5314 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:54.493031 6707 raft_consensus.cc:2726] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:11:54.493265 6811 consensus_meta.cc:287] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.903s user 0.012s sys 0.018s
I20260929 14:11:54.472522 6562 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.542080 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:54.436993 6731 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.546977 6562 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:54.548722 5199 raft_consensus.cc:1226] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:54.482023 6867 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.485045 6930 raft_consensus.cc:3037] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:54.472667 6844 consensus_queue.cc:1046] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260929 14:11:54.473562 5901 raft_consensus.cc:3037] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
W20260929 14:11:54.495194 5273 consensus_meta.cc:287] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 1.170s user 0.001s sys 0.013s
I20260929 14:11:54.495023 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:54.495373 6823 consensus_meta.cc:287] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.940s user 0.020s sys 0.013s
I20260929 14:11:54.495728 6906 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:11:54.532234 6493 raft_consensus.cc:2781] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:11:54.376365 6629 raft_consensus.cc:3037] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:54.481197 5196 raft_consensus.cc:1226] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:54.504494 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.504927 5904 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:11:54.454169 6874 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.505437 5272 raft_consensus.cc:1226] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260929 14:11:54.505813 5871 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:54.451885 6909 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.431125 6905 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:54.506907 6674 consensus_queue.cc:1046] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:11:54.507246 6635 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:54.509574 6907 raft_consensus.cc:2726] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:11:54.512611 6913 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:54.596367 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:54.475570 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.597477 5270 raft_consensus.cc:3037] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:54.599426 6867 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:54.600488 6930 raft_consensus.cc:2726] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:54.492478 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:54.461014 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:54.511890 5195 rpcz_store.cc:276] 0929 14:11:52.988044 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:52.994811 (+ 6767us) service_pool.cc:227] Handling call
0929 14:11:54.511321 (+1516510us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b5400145fd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:54.511431 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3330870016}
I20260929 14:11:54.601197 5268 raft_consensus.cc:2454] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:11:54.601977 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:54.602466 5200 raft_consensus.cc:1261] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:11:54.592649 5273 raft_consensus.cc:2396] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260929 14:11:54.602747 5195 raft_consensus.cc:2454] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
W20260929 14:11:54.603307 5273 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1631) took 1868 ms (client timeout 2159 ms). Trace:
W20260929 14:11:54.603709 5273 rpcz_store.cc:269] 0929 14:11:52.734821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:52.856366 (+121545us) service_pool.cc:227] Handling call
0929 14:11:53.056181 (+199815us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b58001e5498. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0dbce 000000000034aea9 00007fbda9939a30 00007fbda99ff7b4 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:11:54.603223 (+1547042us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27277696}
I20260929 14:11:54.526594 5314 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:54.542047 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:54.605655 6731 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.606033 5347 raft_consensus.cc:2354] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:11:54.605929 6928 raft_consensus.cc:3032] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:54.542564 5201 raft_consensus.cc:1226] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:54.542701 6811 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:54.607829 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 69248.7us
I20260929 14:11:54.608314 6811 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:54.608533 5316 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:54.543119 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:54.551257 5345 raft_consensus.cc:1226] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:54.605692 5900 raft_consensus.cc:1226] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:54.595609 6874 raft_consensus.cc:2781] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:54.609956 6874 raft_consensus.cc:686] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:54.601234 5342 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:54.553696 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:54.611434 5198 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:11:54.537360 6718 consensus_meta.cc:287] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.982s user 0.014s sys 0.022s
I20260929 14:11:54.482954 6879 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.538728 6868 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:54.629876 6868 raft_consensus.cc:686] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:54.498713 6736 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:54.565366 6921 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:54.527186 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 126268us
I20260929 14:11:54.553094 6686 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:54.641664 6686 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.643534 6921 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:54.644305 6686 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:54.646526 5084 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet f84af62485a4405f887bb8675221641e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica c5ec116e39144881bdf1f2cd3e923cf5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:54.649498 5350 raft_consensus.cc:1226] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:54.649963 5898 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260929 14:11:54.650655 5350 raft_consensus.cc:1226] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:54.650972 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.651630 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.652136 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.653669 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.654002 6642 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 14:11:54.652756 5273 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:54.574316 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.662482 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.663237 5275 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:54.663805 5905 raft_consensus.cc:2454] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:54.664552 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:11:54.590183 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 103873us
I20260929 14:11:54.518218 6619 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.665170 5195 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:54.670925 5197 raft_consensus.cc:1226] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:54.670639 5194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37270 (request call id 144) took 1271 ms. Trace:
I20260929 14:11:54.674317 5194 rpcz_store.cc:276] 0929 14:11:53.399576 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:53.544890 (+145314us) service_pool.cc:227] Handling call
0929 14:11:54.670445 (+1125555us) spinlock_profiling.cc:243] Waited 736 ms on lock 0x7b540031e054. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:54.670583 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1693198720}
I20260929 14:11:54.675597 6918 raft_consensus.cc:3032] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:54.676126 6918 raft_consensus.cc:729] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:54.652756 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.677699 5274 raft_consensus.cc:2354] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5.
I20260929 14:11:54.678452 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.678984 5274 raft_consensus.cc:2362] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3. Re-sending same reply.
I20260929 14:11:54.680517 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.681157 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.655220 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.682073 5268 raft_consensus.cc:2362] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3. Re-sending same reply.
I20260929 14:11:54.568363 6493 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:54.595010 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.605341 6587 raft_consensus.cc:2741] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260929 14:11:54.606508 6928 raft_consensus.cc:729] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
W20260929 14:11:54.613379 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 86894.3us
I20260929 14:11:54.685884 6493 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:54.686189 5314 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:54.685685 5097 catalog_manager.cc:5495] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: config changed from index 3 to 4, VOTER c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) evicted. New cstate: current_term: 1 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:54.689082 6811 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:54.688511 5097 catalog_manager.cc:5495] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:54.671247 5343 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:54.693781 6867 raft_consensus.cc:2781] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:54.694830 6867 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:54.685722 6928 consensus_queue.cc:260] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.695215 6867 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:54.695652 6928 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:54.614913 5270 raft_consensus.cc:2454] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:11:54.615980 5316 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:54.616403 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:54.697767 5196 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:54.571974 6732 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.617733 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:54.618185 5902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:54.709653 5902 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:11:54.710250 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 142491us
I20260929 14:11:54.712641 5195 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:11:54.620206 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:54.622295 6659 raft_consensus.cc:3032] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:54.622745 5347 raft_consensus.cc:1226] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:54.623028 6812 raft_consensus.cc:2781] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:11:54.623338 6916 raft_consensus.cc:3032] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:54.623556 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:54.615509 6874 consensus_queue.cc:237] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.588690 6583 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:54.631814 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.722298 5275 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:11:54.722621 5269 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:54.592512 6924 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:11:54.638176 5239 leader_election.cc:398] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:11:54.652907 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.557413 6900 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:54.658445 5342 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:11:54.562145 6823 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.685492 5271 raft_consensus.cc:595] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
W20260929 14:11:54.665053 6934 log.cc:924] Time spent T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.318s user 0.002s sys 0.000s
I20260929 14:11:54.570371 6915 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.741514 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.742544 5270 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:54.719403 6916 raft_consensus.cc:729] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:54.699656 5198 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:11:54.715124 6906 consensus_queue.cc:1046] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
I20260929 14:11:54.701658 6619 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:54.668179 5899 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45232 (request call id 293) took 1620 ms. Trace:
I20260929 14:11:54.666572 5901 raft_consensus.cc:2454] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
W20260929 14:11:54.705706 6947 log.cc:924] Time spent T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.112s user 0.000s sys 0.001s
I20260929 14:11:54.705657 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.709087 5196 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:11:54.751768 5270 raft_consensus.cc:2396] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260929 14:11:54.619949 5351 raft_consensus.cc:1226] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:54.702860 5904 raft_consensus.cc:2379] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 2: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:11:54.714524 5167 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:54.738730 6732 consensus_queue.cc:1046] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260929 14:11:54.718781 5871 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:54.718175 5096 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52726 (request call id 225) took 8269 ms. Trace:
I20260929 14:11:54.753968 6872 raft_consensus.cc:2781] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election won for term 3
W20260929 14:11:54.720137 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 104234us
I20260929 14:11:54.723325 5873 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:54.754467 6872 raft_consensus.cc:686] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:54.754534 6867 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.732095 5239 leader_election.cc:401] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:11:54.755867 6940 raft_consensus.cc:2726] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:54.755534 6872 consensus_queue.cc:237] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:54.756138 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 138713us
I20260929 14:11:54.756655 6867 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:54.754534 5871 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:54.758000 6834 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:54.758260 6935 raft_consensus.cc:2781] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:54.758818 6935 raft_consensus.cc:686] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:54.759425 6587 consensus_queue.cc:1046] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:11:54.759684 6935 consensus_queue.cc:237] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.760476 6587 consensus_queue.cc:1046] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:54.761475 6936 raft_consensus.cc:3032] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:54.761603 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:54.762518 5348 raft_consensus.cc:2379] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 2: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:54.733629 6812 raft_consensus.cc:686] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:54.735745 6917 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:11:54.716086 6918 consensus_queue.cc:260] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.735917 5905 raft_consensus.cc:1226] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:54.630806 6868 consensus_queue.cc:237] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.737135 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.775189 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:54.775799 5276 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:54.776098 6732 raft_consensus.cc:3032] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:54.776537 6732 raft_consensus.cc:729] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:54.776650 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.777858 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.734885 5164 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:54.738335 5271 raft_consensus.cc:603] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:11:54.692888 6493 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.739125 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.717365 6659 raft_consensus.cc:729] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:54.747066 6912 raft_consensus.cc:3032] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:11:54.736595 6900 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.717288 6642 consensus_queue.cc:1046] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.026s
I20260929 14:11:54.749197 5346 raft_consensus.cc:1226] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
W20260929 14:11:54.750074 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 58741.7us
I20260929 14:11:54.747457 5899 rpcz_store.cc:276] 0929 14:11:53.048127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:53.048327 (+ 200us) service_pool.cc:227] Handling call
0929 14:11:53.048739 (+ 412us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:54.466433 (+1417694us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:54.667842 (+201409us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54000cbcd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:54.668070 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3100912512}
I20260929 14:11:54.618584 6718 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.798333 6736 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:54.761842 6936 raft_consensus.cc:729] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:54.758076 5192 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:54.735157 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.764436 5316 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:54.802495 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.803058 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.765590 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.765640 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.786334 5274 raft_consensus.cc:2396] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260929 14:11:54.790432 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.805617 5268 raft_consensus.cc:2379] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:54.806473 6936 consensus_queue.cc:260] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.807858 6936 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:54.785745 6812 consensus_queue.cc:237] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:54.786721 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 72307.1us
I20260929 14:11:54.811687 5167 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:54.787225 6907 raft_consensus.cc:2781] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:54.754268 5096 rpcz_store.cc:276] 0929 14:11:46.449110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:46.449276 (+ 166us) service_pool.cc:227] Handling call
0929 14:11:54.718121 (+8268845us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:11:53.033069 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:11:53.033971 (+ 902us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:11:53.034033 (+ 62us) write_op.cc:435] Acquired schema lock
0929 14:11:53.034060 (+ 27us) tablet.cc:660] Decoding operations
0929 14:11:53.035062 (+ 1002us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:11:53.035201 (+ 139us) write_op.cc:641] Partition lock acquired for write op
0929 14:11:53.035259 (+ 58us) tablet.cc:683] Acquiring locks for 5 operations
0929 14:11:53.036102 (+ 843us) tablet.cc:699] Row locks acquired
0929 14:11:53.036148 (+ 46us) write_op.cc:260] PREPARE: finished
0929 14:11:53.036361 (+ 213us) write_op.cc:270] Start()
0929 14:11:53.036539 (+ 178us) write_op.cc:276] Timestamp: P: 1790691113036287 usec, L: 0
0929 14:11:53.036588 (+ 49us) op_driver.cc:331] REPLICATION: starting
0929 14:11:53.149627 (+113039us) log.cc:842] Serialized 2074 byte log entry
0929 14:11:53.869930 (+720303us) op_driver.cc:444] REPLICATION: finished
0929 14:11:54.208625 (+338695us) write_op.cc:301] APPLY: starting
0929 14:11:54.208736 (+ 111us) tablet.cc:1365] starting BulkCheckPresence
0929 14:11:54.214034 (+ 5298us) tablet.cc:1368] finished BulkCheckPresence
0929 14:11:54.214192 (+ 158us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:11:54.241143 (+ 26951us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:11:54.246981 (+ 5838us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0929 14:11:54.282982 (+ 36001us) write_op.cc:312] APPLY: finished
0929 14:11:54.350097 (+ 67115us) log.cc:842] Serialized 47 byte log entry
0929 14:11:54.350396 (+ 299us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:54.350497 (+ 101us) write_op.cc:454] Released schema lock
0929 14:11:54.350569 (+ 72us) write_op.cc:341] FINISH: Updating metrics
0929 14:11:54.350846 (+ 277us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:54.350872 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1251280,"reported_tablets":15,"child_traces":[["op",{"apply.queue_time_us":306213,"apply.run_cpu_time_us":4442,"apply.run_wall_time_us":164774,"num_ops":5,"prepare.queue_time_us":146137,"prepare.run_cpu_time_us":53700,"prepare.run_wall_time_us":375004,"raft.queue_time_us":309289,"raft.run_cpu_time_us":50852,"raft.run_wall_time_us":231161,"replication_time_us":833239,"thread_start_us":541434,"threads_started":4,"wal-append.queue_time_us":239209}]]}
I20260929 14:11:54.800102 5344 raft_consensus.cc:1226] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260929 14:11:54.814465 6867 consensus_queue.cc:1035] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:54.816555 6920 raft_consensus.cc:3032] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:54.816920 6920 raft_consensus.cc:729] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:54.817909 5314 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:54.817747 6920 consensus_queue.cc:260] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.818998 6920 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:54.788108 6906 raft_consensus.cc:2781] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:54.803104 5271 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:54.801687 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.766979 6587 raft_consensus.cc:3032] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:54.825104 6587 raft_consensus.cc:729] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:54.812196 6736 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:54.826282 6918 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:54.826185 6587 consensus_queue.cc:260] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.788558 6912 raft_consensus.cc:729] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:54.827760 6587 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:54.788301 6493 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:54.827916 6912 consensus_queue.cc:260] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.829231 6912 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:11:54.804451 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:54.778084 6917 raft_consensus.cc:686] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:54.809010 5316 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:54.809535 5270 raft_consensus.cc:1226] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:54.777549 6732 consensus_queue.cc:260] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.810684 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:54.811874 5272 raft_consensus.cc:2379] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
W20260929 14:11:54.787649 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 52869.2us
I20260929 14:11:54.812909 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:54.819579 6661 raft_consensus.cc:2781] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:54.821883 6583 consensus_queue.cc:1035] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:54.822256 6915 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:54.808799 6642 consensus_queue.cc:1225] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer d886fb1eb0474d2d9f7410c4013c45e4 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260929 14:11:54.823001 5276 raft_consensus.cc:2379] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 2: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:54.821974 6811 raft_consensus.cc:2726] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:11:54.823099 6907 raft_consensus.cc:686] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:54.823576 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:54.824440 5902 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:11:54.824606 5268 raft_consensus.cc:1226] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:54.833869 5314 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:54.836262 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:54.824962 6906 raft_consensus.cc:686] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:54.862167 6907 consensus_queue.cc:237] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.863004 6906 consensus_queue.cc:237] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.809334 6916 consensus_queue.cc:260] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.837196 6938 consensus_queue.cc:1046] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260929 14:11:54.838893 6867 raft_consensus.cc:2781] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:54.839797 6917 consensus_queue.cc:237] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.835029 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:54.841192 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.844595 5349 raft_consensus.cc:2379] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 3: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:54.873998 5345 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:54.871760 6476 raft_consensus.cc:2781] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:54.847316 5275 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:48196
I20260929 14:11:54.799750 6718 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:54.850334 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.882613 5902 raft_consensus.cc:1226] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260929 14:11:54.883196 5316 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:54.872262 6675 consensus_queue.cc:1159] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "675b77a0a88a4bf08134281fc0cd949e" responder_term: 3 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260929 14:11:54.852504 6661 raft_consensus.cc:686] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:54.862103 5344 raft_consensus.cc:2379] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 4: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:54.867851 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.841094 6845 consensus_queue.cc:1046] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260929 14:11:54.840706 6921 consensus_queue.cc:1046] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260929 14:11:54.846150 5164 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:54.884231 5316 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:54.857957 5346 raft_consensus.cc:2379] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 2: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:54.870488 5276 raft_consensus.cc:1261] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:11:54.869043 6867 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 14:11:54.902966 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 56952.4us
I20260929 14:11:54.873910 6732 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:54.874575 6916 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:54.911193 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.911250 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.916397 5272 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:54.917290 5342 raft_consensus.cc:1226] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:54.917553 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.922894 5348 raft_consensus.cc:2454] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:11:54.915910 6674 consensus_queue.cc:1046] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260929 14:11:54.912606 6867 raft_consensus.cc:3037] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:54.909804 6735 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:11:54.930560 6476 raft_consensus.cc:686] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
W20260929 14:11:54.939683 5165 leader_election.cc:334] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.529s (SENT)
I20260929 14:11:54.900617 6823 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:54.936702 5272 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:11:54.942108 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.942791 5272 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:54.912990 6741 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election won for term 5
W20260929 14:11:54.947690 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 63555.8us
W20260929 14:11:54.949095 5165 leader_election.cc:334] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.542s (SENT)
I20260929 14:11:54.949965 5316 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:54.932581 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.950656 5165 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:54.961587 6931 consensus_meta.cc:287] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.535s user 0.004s sys 0.006s
I20260929 14:11:54.961833 6587 raft_consensus.cc:3032] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:54.962299 6587 raft_consensus.cc:729] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:54.962142 6931 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.963138 6587 consensus_queue.cc:260] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:54.964330 6587 raft_consensus.cc:3037] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:54.900393 5870 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:54.965677 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 85269.1us
I20260929 14:11:54.965756 6924 raft_consensus.cc:2726] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:11:54.966589 5870 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:54.968214 5870 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:54.969537 6900 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:54.916594 6874 consensus_queue.cc:1046] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260929 14:11:54.977290 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.978708 5276 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:54.979408 5271 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:11:54.980015 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.981117 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.982882 5276 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:54.984767 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:54.987080 5270 raft_consensus.cc:2379] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:54.989651 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.920606 6661 consensus_queue.cc:237] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:54.995496 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.993978 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:54.992686 5316 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:54.995448 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:11:54.972697 6952 log.cc:924] Time spent T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.067s user 0.000s sys 0.001s
I20260929 14:11:54.996338 5314 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:54.954530 6741 raft_consensus.cc:686] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:54.996856 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.997819 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:54.998339 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:55.014868 5199 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:54.999805 5314 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:54.954870 6823 raft_consensus.cc:2781] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:11:54.976794 6834 raft_consensus.cc:2726] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:11:55.018152 5195 raft_consensus.cc:2379] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 2: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:54.981979 6924 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:55.001016 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:54.977097 6941 raft_consensus.cc:2726] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:54.971882 6718 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:54.984593 6914 consensus_meta.cc:287] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.603s user 0.012s sys 0.000s
I20260929 14:11:55.001334 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:55.001603 6635 raft_consensus.cc:2781] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:11:54.995079 6868 raft_consensus.cc:2781] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:55.002574 6707 raft_consensus.cc:2781] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:55.002681 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:55.001060 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:54.990819 6751 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:55.005538 6900 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:55.006403 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:55.007392 5348 raft_consensus.cc:1261] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:11:54.988518 6735 raft_consensus.cc:686] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:55.007597 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.976701 5274 raft_consensus.cc:2454] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:55.009305 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:54.987836 6931 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:55.078549 6718 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.011215 6476 consensus_queue.cc:237] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.015854 5345 raft_consensus.cc:2454] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:11:55.015928 5271 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:54.942615 6907 consensus_queue.cc:1046] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260929 14:11:54.988698 6928 raft_consensus.cc:2726] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:55.049609 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:55.050598 6946 consensus_queue.cc:1046] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:11:55.082360 5270 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:55.090034 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:55.089903 6735 consensus_queue.cc:237] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.092599 6868 raft_consensus.cc:686] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:55.113477 5274 raft_consensus.cc:2362] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5. Re-sending same reply.
I20260929 14:11:55.093184 5269 raft_consensus.cc:2379] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 3: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:11:55.114151 6868 consensus_queue.cc:237] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.114748 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:55.115327 5269 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:55.115468 5899 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:11:55.092746 6900 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.093880 5342 raft_consensus.cc:1226] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:55.114476 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:55.075351 6822 raft_consensus.cc:2726] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260929 14:11:55.099752 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 71013.4us
I20260929 14:11:55.120637 6550 consensus_queue.cc:1035] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:55.077790 5192 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 3
I20260929 14:11:55.120873 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:55.124799 6907 raft_consensus.cc:3032] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:55.121026 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:55.126103 6635 raft_consensus.cc:686] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:55.131475 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:55.081753 5905 raft_consensus.cc:1226] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:55.124650 6914 consensus_queue.cc:1046] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:55.139564 6936 raft_consensus.cc:3032] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:55.079138 6823 raft_consensus.cc:686] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:55.139994 6936 raft_consensus.cc:729] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:55.140844 6936 consensus_queue.cc:260] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.140806 6823 consensus_queue.cc:237] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.130525 6907 raft_consensus.cc:729] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:55.144512 6907 consensus_queue.cc:260] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.126956 6946 consensus_queue.cc:1225] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer d886fb1eb0474d2d9f7410c4013c45e4 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260929 14:11:55.146044 6907 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:55.134606 6741 consensus_queue.cc:237] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.087867 6707 raft_consensus.cc:686] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:55.149761 6707 consensus_queue.cc:237] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:55.150823 5269 raft_consensus.cc:2354] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3.
I20260929 14:11:55.150269 6659 consensus_queue.cc:260] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:55.151657 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:55.095350 5268 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:55.181591 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:55.182106 6642 raft_consensus.cc:3032] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:55.182596 6642 raft_consensus.cc:729] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:55.182634 5274 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:55.183676 6642 consensus_queue.cc:260] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:55.134513 6635 consensus_queue.cc:237] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:55.188056 5275 raft_consensus.cc:595] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:55.191008 5347 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:11:55.191658 5347 raft_consensus.cc:595] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:55.192020 5347 raft_consensus.cc:603] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:11:55.193579 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260929 14:11:55.164394 5870 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.195132 6913 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:55.196609 6909 raft_consensus.cc:3032] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:55.197078 6909 raft_consensus.cc:729] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:55.198231 6909 consensus_queue.cc:260] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.199945 6654 raft_consensus.cc:3032] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:55.200354 6915 raft_consensus.cc:3032] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:55.200866 6915 raft_consensus.cc:729] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:55.204282 6909 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:55.200383 6654 raft_consensus.cc:729] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:55.204901 6915 consensus_queue.cc:260] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.214666 6654 consensus_queue.cc:260] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:55.225360 5350 raft_consensus.cc:1226] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5. Ops: []
I20260929 14:11:55.229314 6946 consensus_queue.cc:1046] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:55.230028 6946 consensus_queue.cc:1225] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer d886fb1eb0474d2d9f7410c4013c45e4 log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260929 14:11:55.230535 6946 raft_consensus.cc:3032] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:11:55.230907 6946 raft_consensus.cc:729] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:55.231951 6946 consensus_queue.cc:260] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.233316 6946 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:55.245373 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:11:55.245623 6936 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:55.232439 5275 raft_consensus.cc:3032] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:55.251705 5275 raft_consensus.cc:729] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:55.252828 5275 consensus_queue.cc:260] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.254750 5275 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:55.152709 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:11:55.269052 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 269278us
I20260929 14:11:55.269284 5269 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:55.269891 5314 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:11:55.268893 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 319894us
I20260929 14:11:55.271516 5314 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:55.275688 5271 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:55.298928 5901 raft_consensus.cc:1226] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:55.299279 5199 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:55.276111 5314 leader_election.cc:398] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3.
I20260929 14:11:55.302788 5314 leader_election.cc:401] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260929 14:11:55.277204 5276 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:55.305197 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:55.286264 5270 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:55.290280 5272 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
W20260929 14:11:55.290980 5165 leader_election.cc:334] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.568s (SENT)
I20260929 14:11:55.311151 5165 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.312107 5270 raft_consensus.cc:1261] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:55.314479 5272 raft_consensus.cc:1261] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260929 14:11:55.315433 5242 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.306636 5273 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:55.087965 6751 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.316442 5316 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:55.318478 5275 raft_consensus.cc:1261] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:11:55.320407 5314 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:55.321527 6916 raft_consensus.cc:2726] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:55.292375 6812 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:55.282169 6493 raft_consensus.cc:2726] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:55.323992 6717 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
W20260929 14:11:55.293094 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 212544us
I20260929 14:11:55.308161 6642 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:55.308589 6915 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:55.325999 6935 consensus_queue.cc:1035] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:11:55.328872 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:55.314432 6654 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:55.307137 6900 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:55.317003 6926 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3.
I20260929 14:11:55.381872 5905 raft_consensus.cc:1226] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:55.390019 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:55.330487 5242 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.331130 5347 raft_consensus.cc:2454] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:55.335244 5267 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:55.334993 5314 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:55.316776 6659 raft_consensus.cc:3037] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:55.324158 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
W20260929 14:11:55.339310 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 48362.3us
I20260929 14:11:55.399371 5342 raft_consensus.cc:1226] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:55.401926 6900 raft_consensus.cc:2726] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260929 14:11:55.401252 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 70826.2us
I20260929 14:11:55.404361 5242 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:55.405092 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 63643.2us
I20260929 14:11:55.405865 5242 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.406536 6900 raft_consensus.cc:2726] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:55.407061 6900 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:55.407696 6907 consensus_queue.cc:1046] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:55.341807 6812 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:55.358264 6751 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:55.360016 6718 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:11:55.360775 6736 consensus_meta.cc:287] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.535s user 0.015s sys 0.002s
I20260929 14:11:55.326954 6872 consensus_queue.cc:1035] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260929 14:11:55.363987 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:11:55.329246 6845 raft_consensus.cc:2726] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:11:55.383512 5345 raft_consensus.cc:1261] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:11:55.341298 5195 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:55.408880 5239 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:55.410899 5165 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.442533 5165 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.355157 6717 raft_consensus.cc:686] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:55.411242 5242 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.411541 6731 raft_consensus.cc:3032] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Stepping down as leader of term 3
W20260929 14:11:55.412544 6920 consensus_meta.cc:287] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.593s user 0.011s sys 0.010s
I20260929 14:11:55.413305 5347 raft_consensus.cc:1261] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260929 14:11:55.417809 5267 raft_consensus.cc:1261] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 14:11:55.448130 6922 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:55.448249 5097 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44250 (request call id 124) took 7562 ms. Trace:
I20260929 14:11:55.425108 5905 raft_consensus.cc:1261] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:11:55.448735 6845 raft_consensus.cc:2781] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:55.449117 6845 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:55.429749 5270 raft_consensus.cc:1261] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260929 14:11:55.449472 6845 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:55.429926 5342 raft_consensus.cc:1261] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 14:11:55.449904 6675 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:55.421995 5199 raft_consensus.cc:2454] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:55.450510 6675 consensus_queue.cc:1035] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:11:55.451037 5264 tablet_service.cc:1514] Processing DeleteTablet for tablet d653defbcc7e47ca89b35b1383e4e2c8 with delete_type TABLET_DATA_TOMBSTONED (TS c5ec116e39144881bdf1f2cd3e923cf5 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:50768
I20260929 14:11:55.454746 5165 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:55.446825 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 111934us
I20260929 14:11:55.447541 6731 raft_consensus.cc:729] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:55.425181 6736 raft_consensus.cc:2726] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260929 14:11:55.461650 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 50451.4us
I20260929 14:11:55.457760 5314 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:55.462955 5242 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.459841 5165 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.446003 5316 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:55.454983 5269 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:55.468434 5350 raft_consensus.cc:1226] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:55.469748 5314 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:55.470207 5199 raft_consensus.cc:1261] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:11:55.473790 6751 raft_consensus.cc:2781] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:55.474203 6751 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:55.474529 6751 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:55.474785 5242 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.475548 6900 raft_consensus.cc:3037] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:55.476224 5242 leader_election.cc:398] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:11:55.476547 5242 leader_election.cc:401] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260929 14:11:55.474815 6926 raft_consensus.cc:2781] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:11:55.477170 6926 raft_consensus.cc:686] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:55.473814 6731 consensus_queue.cc:260] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.465510 6936 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:55.478502 6731 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:55.478790 6936 raft_consensus.cc:2726] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:11:55.475612 5904 raft_consensus.cc:1261] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260929 14:11:55.478101 6926 consensus_queue.cc:237] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.453143 6928 raft_consensus.cc:2726] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:11:55.482551 5242 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.462523 6907 raft_consensus.cc:2726] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:55.463655 6812 raft_consensus.cc:686] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:55.506160 6812 consensus_queue.cc:237] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.506834 5192 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:55.466982 6909 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260929 14:11:55.510035 5274 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:11:55.510289 6717 consensus_queue.cc:237] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.480931 6946 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:55.465472 6941 consensus_queue.cc:1035] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.052s
I20260929 14:11:55.517630 5242 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.519003 5242 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.520147 6914 raft_consensus.cc:2741] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260929 14:11:55.520705 6914 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:11:55.521016 6914 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:55.521381 6914 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:11:55.521947 6946 consensus_queue.cc:1046] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:11:55.522884 6834 raft_consensus.cc:3032] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:55.523216 6834 raft_consensus.cc:729] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:55.472625 6476 raft_consensus.cc:2781] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:55.497632 5316 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:55.465621 6922 raft_consensus.cc:686] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:55.509847 6936 consensus_queue.cc:1046] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:11:55.448768 5097 rpcz_store.cc:276] 0929 14:11:47.886200 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:47.886371 (+ 171us) service_pool.cc:227] Handling call
0929 14:11:55.448179 (+7561808us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:11:54.848282 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:11:54.969388 (+121106us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:11:54.969423 (+ 35us) write_op.cc:435] Acquired schema lock
0929 14:11:54.969439 (+ 16us) tablet.cc:660] Decoding operations
0929 14:11:54.969897 (+ 458us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:11:54.969967 (+ 70us) write_op.cc:641] Partition lock acquired for write op
0929 14:11:54.969993 (+ 26us) tablet.cc:683] Acquiring locks for 2 operations
0929 14:11:54.970292 (+ 299us) tablet.cc:699] Row locks acquired
0929 14:11:54.970312 (+ 20us) write_op.cc:260] PREPARE: finished
0929 14:11:54.970452 (+ 140us) write_op.cc:270] Start()
0929 14:11:54.970561 (+ 109us) write_op.cc:276] Timestamp: P: 1790691114970410 usec, L: 0
0929 14:11:54.970583 (+ 22us) op_driver.cc:331] REPLICATION: starting
0929 14:11:55.003371 (+ 32788us) log.cc:842] Serialized 913 byte log entry
0929 14:11:55.243951 (+240580us) op_driver.cc:444] REPLICATION: finished
0929 14:11:55.425485 (+181534us) write_op.cc:301] APPLY: starting
0929 14:11:55.425578 (+ 93us) tablet.cc:1365] starting BulkCheckPresence
0929 14:11:55.425687 (+ 109us) tablet.cc:1368] finished BulkCheckPresence
0929 14:11:55.431202 (+ 5515us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:11:55.431569 (+ 367us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:11:55.431719 (+ 150us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0929 14:11:55.431760 (+ 41us) write_op.cc:312] APPLY: finished
0929 14:11:55.431951 (+ 191us) log.cc:842] Serialized 29 byte log entry
0929 14:11:55.432106 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:55.432189 (+ 83us) write_op.cc:454] Released schema lock
0929 14:11:55.432223 (+ 34us) write_op.cc:341] FINISH: Updating metrics
0929 14:11:55.432489 (+ 266us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:55.432520 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":221411,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":181292,"apply.run_cpu_time_us":2144,"apply.run_wall_time_us":7279,"num_ops":2,"prepare.queue_time_us":103106,"prepare.run_cpu_time_us":4092,"prepare.run_wall_time_us":158172,"replication_time_us":273280,"thread_start_us":181878,"threads_started":2}]]}
I20260929 14:11:55.500222 5165 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.524449 5242 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.534322 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:55.485766 6675 raft_consensus.cc:2781] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:55.517011 5098 catalog_manager.cc:5495] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:11:55.537191 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:55.549876 5098 catalog_manager.cc:5495] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:55.548071 5270 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:55.491709 6587 raft_consensus.cc:2726] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:11:55.465730 6874 consensus_queue.cc:1035] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
W20260929 14:11:55.525381 6969 log.cc:924] Time spent T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.086s user 0.001s sys 0.000s
I20260929 14:11:55.551730 5098 catalog_manager.cc:5495] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 1 to 3, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 3 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
W20260929 14:11:55.545828 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 48278.1us
I20260929 14:11:55.557430 5342 raft_consensus.cc:1261] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260929 14:11:55.559154 5267 raft_consensus.cc:1261] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260929 14:11:55.566659 5267 pending_rounds.cc:76] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5: Aborting all ops after (but not including) 1
I20260929 14:11:55.567006 5267 pending_rounds.cc:97] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260929 14:11:55.567307 5267 raft_consensus.cc:2866] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260929 14:11:55.573939 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 73791.3us
I20260929 14:11:55.578598 5165 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.560575 6907 raft_consensus.cc:2726] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:11:55.570333 5098 catalog_manager.cc:5495] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:11:55.563320 6629 raft_consensus.cc:2781] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:55.552639 6909 raft_consensus.cc:2726] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:55.600580 5098 catalog_manager.cc:5495] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 1 to 2, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 2 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:55.552486 6635 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:55.564224 6476 raft_consensus.cc:686] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:55.538841 6403 consensus_queue.cc:1035] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260929 14:11:55.604063 5165 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:55.583287 6924 consensus_meta.cc:287] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.507s user 0.010s sys 0.006s
I20260929 14:11:55.553804 6935 consensus_queue.cc:1035] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.053s
W20260929 14:11:55.586505 6867 consensus_meta.cc:287] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.640s user 0.012s sys 0.008s
I20260929 14:11:55.576265 6675 raft_consensus.cc:686] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:55.602661 6629 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 14:11:55.631174 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 48458.3us
W20260929 14:11:55.635457 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 66999us
I20260929 14:11:55.600198 5194 raft_consensus.cc:1261] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 14:11:55.635317 5098 catalog_manager.cc:5495] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260929 14:11:55.573837 6941 consensus_queue.cc:1035] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.050s
I20260929 14:11:55.638659 6909 consensus_queue.cc:1046] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.010s
W20260929 14:11:55.649811 6975 log.cc:924] Time spent T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
W20260929 14:11:55.631250 6912 consensus_meta.cc:287] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.802s user 0.014s sys 0.008s
I20260929 14:11:55.640259 6924 raft_consensus.cc:2726] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:55.667623 5270 raft_consensus.cc:1261] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260929 14:11:55.668025 5270 pending_rounds.cc:76] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5: Aborting all ops after (but not including) 5
I20260929 14:11:55.668278 5270 pending_rounds.cc:97] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260929 14:11:55.668510 5270 raft_consensus.cc:2866] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:11:55.614220 6661 raft_consensus.cc:2726] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:11:55.655707 5098 catalog_manager.cc:5495] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 2 to 4, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260929 14:11:55.669781 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 65828.2us
I20260929 14:11:55.671381 5165 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:55.673815 5198 raft_consensus.cc:1261] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
I20260929 14:11:55.665086 5267 raft_consensus.cc:1261] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260929 14:11:55.677598 5267 pending_rounds.cc:76] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5: Aborting all ops after (but not including) 1
I20260929 14:11:55.677922 5267 pending_rounds.cc:97] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5: Aborting uncommitted NO_OP operation due to leader change: 4.2
I20260929 14:11:55.678227 5267 raft_consensus.cc:2866] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:11:55.665377 6909 raft_consensus.cc:3032] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:55.667841 5268 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:55.683372 5098 catalog_manager.cc:5495] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:11:55.568799 6834 consensus_queue.cc:260] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:55.690213 5276 raft_consensus.cc:1226] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:55.644677 6550 raft_consensus.cc:2781] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:55.602288 6654 consensus_queue.cc:1035] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260929 14:11:55.654234 6629 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
W20260929 14:11:55.713277 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 138008us
I20260929 14:11:55.714260 5242 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:55.623769 6635 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:55.719445 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 48850.4us
I20260929 14:11:55.636190 6562 consensus_queue.cc:1035] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20260929 14:11:55.694733 6661 raft_consensus.cc:2726] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:55.728931 5242 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:55.685405 6909 raft_consensus.cc:729] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:55.645490 6867 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:55.732370 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 54642.4us
I20260929 14:11:55.613675 6922 consensus_queue.cc:237] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.701437 6550 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:55.733960 5314 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:55.737926 5350 raft_consensus.cc:1261] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:55.718799 6834 raft_consensus.cc:3037] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:55.752704 6635 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:55.720785 6912 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:55.754695 6912 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.755059 6587 raft_consensus.cc:3032] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:55.755511 6587 raft_consensus.cc:729] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:55.756657 6912 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:55.744060 6732 consensus_meta.cc:287] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.827s user 0.012s sys 0.006s
I20260929 14:11:55.745543 6909 consensus_queue.cc:260] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:55.737434 5346 raft_consensus.cc:2379] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:55.725052 5899 raft_consensus.cc:1261] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:11:55.737560 6985 raft_consensus.cc:2726] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:55.735302 6915 consensus_queue.cc:1035] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
I20260929 14:11:55.758452 5898 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:11:55.726840 6476 consensus_queue.cc:237] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.752276 6914 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.735659 6811 consensus_queue.cc:1035] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.024s
I20260929 14:11:55.742870 6867 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:55.762566 5267 raft_consensus.cc:1261] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:55.756388 6587 consensus_queue.cc:260] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.765256 5165 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.770535 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:55.791718 5345 raft_consensus.cc:2379] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 3: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:55.739967 6550 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:55.768158 6735 consensus_queue.cc:1035] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260929 14:11:55.784050 6984 raft_consensus.cc:2726] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:55.736369 5904 raft_consensus.cc:1261] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260929 14:11:55.785151 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 51266.6us
I20260929 14:11:55.785832 5242 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:55.793334 6732 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:11:55.798324 5165 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.789680 6909 raft_consensus.cc:3037] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:55.820564 6914 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:55.822013 6990 log.cc:924] Time spent T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.077s user 0.003s sys 0.000s
I20260929 14:11:55.821821 6717 consensus_queue.cc:1035] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:11:55.823014 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:55.816674 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:55.824646 5275 raft_consensus.cc:3037] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:55.790578 6915 raft_consensus.cc:2781] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election won for term 2
W20260929 14:11:55.827325 6980 log.cc:924] Time spent T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20260929 14:11:55.807509 6642 raft_consensus.cc:2726] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:55.835419 5904 raft_consensus.cc:1261] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:55.838760 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:55.824674 5267 raft_consensus.cc:1261] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:11:55.818554 6629 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.818761 6845 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.736922 6675 consensus_queue.cc:237] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:55.844169 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 119792us
I20260929 14:11:55.845641 5165 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.846592 5275 raft_consensus.cc:2454] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:55.856422 5165 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.857088 5199 raft_consensus.cc:1261] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:11:55.860457 5201 raft_consensus.cc:1261] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:11:55.813012 6984 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:11:55.877727 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:55.881924 5198 raft_consensus.cc:3037] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 4
I20260929 14:11:55.882479 5267 raft_consensus.cc:1261] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260929 14:11:55.882755 5272 raft_consensus.cc:1226] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:11:55.879122 5314 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:55.812554 6926 consensus_queue.cc:1035] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.025s
I20260929 14:11:55.887598 5165 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.890178 5273 raft_consensus.cc:2379] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 4: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:55.848377 6900 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:55.893352 5165 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.842599 6915 raft_consensus.cc:686] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:55.877396 6659 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:55.908651 5192 raft_consensus.cc:1261] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260929 14:11:55.909387 6920 raft_consensus.cc:2726] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:11:55.910084 6920 consensus_queue.cc:1035] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:55.911010 6978 tablet_replica.cc:330] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5: stopping tablet replica
I20260929 14:11:55.912091 6978 raft_consensus.cc:2229] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260929 14:11:55.866815 6741 consensus_queue.cc:1035] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
W20260929 14:11:55.915484 6988 log.cc:924] Time spent T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.174s user 0.001s sys 0.000s
I20260929 14:11:55.916531 5198 raft_consensus.cc:2454] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:55.896978 6642 raft_consensus.cc:2726] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:55.906541 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:55.887279 6493 consensus_queue.cc:1035] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.020s
I20260929 14:11:55.912803 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:55.886860 6984 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:55.895566 6476 consensus_queue.cc:1035] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260929 14:11:55.886997 6989 consensus_queue.cc:1046] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260929 14:11:55.887655 6717 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:55.906081 6915 consensus_queue.cc:237] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.918190 5165 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.920320 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:55.889117 6731 raft_consensus.cc:2726] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:11:55.921837 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:55.933948 5276 raft_consensus.cc:2354] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3.
I20260929 14:11:55.917587 6867 raft_consensus.cc:2781] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:55.926136 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:55.915858 6991 consensus_queue.cc:1046] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:11:55.936439 6926 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:55.940624 5165 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.945578 5165 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:55.946683 5165 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.946609 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:55.948220 5165 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.953629 6629 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:55.953975 6916 raft_consensus.cc:2726] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:11:55.958221 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:55.906119 6921 consensus_queue.cc:1035] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.035s
I20260929 14:11:55.927134 6984 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:11:55.963358 5165 leader_election.cc:398] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3.
I20260929 14:11:55.966976 5165 leader_election.cc:401] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:11:55.968410 5165 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:55.974366 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:55.977281 5342 raft_consensus.cc:1261] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:11:55.922151 6751 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:55.978444 5267 raft_consensus.cc:2354] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
W20260929 14:11:55.980417 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 54137us
I20260929 14:11:55.945847 6731 raft_consensus.cc:2741] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260929 14:11:55.987373 5165 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:55.947397 6642 raft_consensus.cc:2726] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:55.981596 6732 raft_consensus.cc:2726] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3.
I20260929 14:11:55.956140 6661 raft_consensus.cc:2726] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:55.941011 6935 consensus_queue.cc:1035] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260929 14:11:55.966338 6916 raft_consensus.cc:2726] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:11:55.988468 5316 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:55.973697 6978 raft_consensus.cc:2258] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 14:11:56.004876 6917 consensus_queue.cc:1035] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:11:55.975281 6867 raft_consensus.cc:686] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:55.941527 6717 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.015064 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:55.963883 6823 consensus_queue.cc:1035] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.046s
I20260929 14:11:55.989317 6928 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:55.989643 6920 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:55.974077 6812 consensus_queue.cc:1035] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260929 14:11:55.946584 6926 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.038195 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:56.043665 5345 raft_consensus.cc:2379] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:56.010842 6642 raft_consensus.cc:2726] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:56.026768 6732 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:56.041777 6845 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:56.007808 6731 consensus_queue.cc:1035] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.038s
I20260929 14:11:56.052410 6928 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:11:56.080436 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 101630us
I20260929 14:11:56.042629 6920 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.091387 6928 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.092099 6751 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:56.092860 6920 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:56.093386 6928 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:56.093645 6550 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:56.095201 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:56.096037 5904 raft_consensus.cc:2454] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:56.096252 6550 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:56.097335 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:56.050920 6476 consensus_queue.cc:1035] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.040s
I20260929 14:11:56.057350 6921 raft_consensus.cc:2726] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:56.100843 5197 raft_consensus.cc:1226] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:56.108405 5198 raft_consensus.cc:1261] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:56.102501 5167 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:56.090114 6659 raft_consensus.cc:2726] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:56.064332 6732 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.143186 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:56.074021 6868 consensus_queue.cc:1035] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
I20260929 14:11:56.136082 6550 raft_consensus.cc:2781] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260929 14:11:56.149627 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 162307us
I20260929 14:11:56.121033 6928 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:56.152740 5165 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:56.154887 5165 leader_election.cc:398] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:11:56.152004 6935 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:56.153942 6550 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:56.154000 6659 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:56.159556 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:56.103257 6867 consensus_queue.cc:237] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:56.166883 6916 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:56.195942 5194 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:56.181540 6935 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:56.202813 6995 log.cc:924] Time spent T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.131s user 0.001s sys 0.000s
I20260929 14:11:56.203454 6935 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:56.179263 6550 raft_consensus.cc:3037] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:56.206246 6717 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:56.206897 6899 log.cc:924] Time spent T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29: Append to log took a long time: real 0.109s user 0.000s sys 0.002s
I20260929 14:11:56.207862 5242 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:56.156652 6928 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:56.219650 6732 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:56.220443 6926 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:56.218243 6978 ts_tablet_manager.cc:1905] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260929 14:11:56.260161 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 180215us
W20260929 14:11:56.274132 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 66331.8us
I20260929 14:11:56.258450 6845 raft_consensus.cc:2781] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:56.258873 7008 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:56.264357 6922 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.011s sys 0.018s
W20260929 14:11:56.297034 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 75197.9us
I20260929 14:11:56.208891 6916 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:56.302728 5314 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:11:56.295424 6922 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.548s user 0.011s sys 0.019s
I20260929 14:11:56.305833 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:56.291203 6675 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:56.308758 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:56.295120 6928 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:56.312436 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:56.306694 5349 raft_consensus.cc:2454] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:56.315095 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:56.301090 5902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:56.319821 5870 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:56.306684 6845 raft_consensus.cc:686] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:56.314043 5164 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:56.342435 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:56.344862 5270 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:56.332909 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:56.296321 7008 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:56.318274 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:56.307863 6984 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.334525 6909 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:56.341302 7004 raft_consensus.cc:2726] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:11:56.374733 5267 raft_consensus.cc:1261] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:56.370266 6977 raft_consensus.cc:2781] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:56.312335 6675 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:56.350080 6926 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:56.368005 6732 raft_consensus.cc:2726] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:56.347443 5902 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:56.409092 5267 raft_consensus.cc:1261] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260929 14:11:56.371019 7027 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:56.376399 5270 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:56.446298 5899 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45252 (request call id 418) took 1167 ms. Trace:
I20260929 14:11:56.441452 6926 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.446707 5899 rpcz_store.cc:276] 0929 14:11:55.278620 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:55.281874 (+ 3254us) service_pool.cc:227] Handling call
0929 14:11:55.323847 (+ 41973us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:55.764419 (+440572us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:56.444448 (+680029us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b540071ded4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:56.444619 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":798350080}
I20260929 14:11:56.379678 6977 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:56.337255 7018 consensus_queue.cc:1035] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.104s
I20260929 14:11:56.448926 5167 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:56.450968 5267 raft_consensus.cc:1261] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:11:56.377188 6909 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:56.457933 6977 raft_consensus.cc:3037] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:56.482618 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:56.453979 5343 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:56.499368 5270 raft_consensus.cc:1226] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:11:56.475970 6834 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.107s
I20260929 14:11:56.481352 5194 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:56.476228 7011 consensus_queue.cc:1035] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.123s
I20260929 14:11:56.601380 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:56.482321 6916 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:56.484256 6731 raft_consensus.cc:2781] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260929 14:11:56.593819 7030 log.cc:924] Time spent T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.117s user 0.001s sys 0.000s
I20260929 14:11:56.628563 5342 raft_consensus.cc:1226] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:56.598479 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:56.481905 6736 consensus_queue.cc:1035] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.144s
I20260929 14:11:56.625264 6659 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:56.442859 7027 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:56.634707 7012 log.cc:924] Time spent T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.325s user 0.000s sys 0.001s
I20260929 14:11:56.638888 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:56.639406 5901 raft_consensus.cc:1261] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260929 14:11:56.643628 7021 log.cc:924] Time spent T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.264s user 0.001s sys 0.000s
W20260929 14:11:56.644291 7016 log.cc:924] Time spent T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.234s user 0.001s sys 0.000s
I20260929 14:11:56.625813 6920 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:11:56.652098 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 353733us
I20260929 14:11:56.629181 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:56.629433 6731 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:56.482146 6845 consensus_queue.cc:237] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.637184 6926 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:56.660279 5270 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:56.661456 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:56.481793 6822 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:11:56.651633 7022 log.cc:924] Time spent T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.310s user 0.001s sys 0.000s
I20260929 14:11:56.665274 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:56.666226 5342 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:56.667635 5164 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:56.667953 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:56.668790 5348 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:56.670058 5164 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:56.670938 6493 raft_consensus.cc:3032] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:11:56.671447 6493 raft_consensus.cc:729] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:56.673173 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:56.673863 5276 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:56.674715 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:56.674839 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:56.675379 5276 raft_consensus.cc:2354] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6.
I20260929 14:11:56.676200 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:56.676888 5276 raft_consensus.cc:2354] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3.
I20260929 14:11:56.676224 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:56.679800 5197 raft_consensus.cc:2354] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:11:56.680668 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:56.682003 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:56.655821 7008 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:56.659137 6920 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.640010 6916 consensus_queue.cc:1046] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260929 14:11:56.685997 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:56.696864 5197 raft_consensus.cc:2454] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:56.698184 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:56.698660 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:56.698961 5269 raft_consensus.cc:2379] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 2: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
W20260929 14:11:56.698259 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 350444us
I20260929 14:11:56.699354 5197 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:56.696571 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:56.700428 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:56.700805 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:56.663849 6731 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:56.672415 6493 consensus_queue.cc:260] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:56.703259 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 265831us
I20260929 14:11:56.706775 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:56.709671 5270 raft_consensus.cc:1261] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:11:56.666571 6675 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:56.707474 5197 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:56.714622 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:56.715435 5276 raft_consensus.cc:2454] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:56.711611 5269 raft_consensus.cc:1226] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:56.710067 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:11:56.650173 6659 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.697365 6909 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:56.724869 5269 raft_consensus.cc:1261] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260929 14:11:56.691123 6834 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:56.693055 6642 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:56.687155 6920 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:56.713654 6654 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:56.720537 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:56.683897 6822 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.733623 6909 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:56.743109 5276 raft_consensus.cc:1261] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:11:56.733232 6642 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:56.738849 6493 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:11:56.729950 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:56.737610 6984 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:56.770714 5904 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
W20260929 14:11:56.773731 6986 log.cc:924] Time spent T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 1.033s user 0.000s sys 0.010s
I20260929 14:11:56.774731 6822 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:56.776031 5347 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47258 (request call id 1465) took 1249 ms. Trace:
I20260929 14:11:56.737780 6834 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:56.759104 6642 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:56.776086 7027 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:56.776690 5347 rpcz_store.cc:276] 0929 14:11:55.526161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:55.526284 (+ 123us) service_pool.cc:227] Handling call
0929 14:11:55.526599 (+ 315us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:55.545026 (+ 18427us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:55.545054 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:55.554637 (+ 9583us) log.cc:842] Serialized 26 byte log entry
0929 14:11:55.658501 (+103864us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:11:55.658545 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:55.658617 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:56.744720 (+1086103us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda46efbee 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551 00007fbda46f8509 00007fbda46f8499
0929 14:11:56.775614 (+ 30894us) raft_consensus.cc:1671] finished
0929 14:11:56.775643 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:56.775934 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18361,"raft.queue_time_us":18822,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":10238,"spinlock_wait_cycles":41802496,"thread_start_us":74454,"threads_started":1,"wal-append.queue_time_us":158390}
I20260929 14:11:56.760071 6909 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:56.753605 6654 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.844316 6741 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:56.766145 6977 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.877496 5314 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:56.853662 6928 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:56.885548 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 320687us
I20260929 14:11:56.887979 5197 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:11:56.891754 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:56.879222 7008 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:56.892789 5197 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:11:56.894313 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:56.891728 6550 consensus_meta.cc:287] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.684s user 0.008s sys 0.004s
I20260929 14:11:56.897579 6659 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:56.905398 7038 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:56.886209 6928 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:56.905884 7038 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.879305 6741 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.907780 7038 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:56.908591 6741 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:56.910441 5242 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:56.902532 6834 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:56.904800 6550 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.901700 5900 raft_consensus.cc:2379] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 3: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:11:56.889185 7032 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:56.911716 5242 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:56.926286 5242 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:56.910873 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:56.927023 5907 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:11:56.914590 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:56.914654 5164 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:56.919616 5200 raft_consensus.cc:1226] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:56.920462 5199 raft_consensus.cc:1226] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 2. Current term is 3. Ops: []
W20260929 14:11:56.923780 5273 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50776 (request call id 1342) took 1389 ms (client timeout 1806 ms). Trace:
W20260929 14:11:56.930337 5273 rpcz_store.cc:269] 0929 14:11:55.533929 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:55.534027 (+ 98us) service_pool.cc:227] Handling call
0929 14:11:56.923540 (+1389513us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b5400321ed4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:56.923713 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":786707968}
I20260929 14:11:56.938916 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:56.911343 6984 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:56.944695 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:56.948604 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:56.901016 5194 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:56.919694 6834 raft_consensus.cc:2726] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:11:56.951002 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:56.965149 5200 raft_consensus.cc:2379] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 2: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
W20260929 14:11:56.928045 6915 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.898s user 0.007s sys 0.004s
I20260929 14:11:56.918591 6718 raft_consensus.cc:2781] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:56.953052 5873 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:56.950398 5199 raft_consensus.cc:2379] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 5: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:56.954780 6550 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:56.955197 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:56.956023 7007 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:56.942973 7032 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:56.898887 7008 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:56.952329 6984 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:56.959201 5165 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:56.986474 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:56.988571 5200 raft_consensus.cc:2454] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:56.988988 5165 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:56.989527 5200 raft_consensus.cc:1226] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:56.990142 5165 leader_election.cc:398] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6.
W20260929 14:11:56.970099 6915 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.940s user 0.007s sys 0.004s
I20260929 14:11:56.938658 6916 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:56.991865 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:56.941102 6659 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:56.978865 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:56.946890 6912 raft_consensus.cc:2741] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election decision vote started in defunct term 1: won
I20260929 14:11:56.981565 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:57.001719 5199 raft_consensus.cc:2379] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:57.003218 5199 raft_consensus.cc:1261] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260929 14:11:56.973632 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 96258.7us
I20260929 14:11:56.990411 5165 leader_election.cc:401] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:11:56.959918 7042 raft_consensus.cc:2726] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260929 14:11:56.991145 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:57.005625 5903 raft_consensus.cc:3037] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:57.001741 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:56.979938 6834 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:56.979300 7027 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:56.985152 7028 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:11:57.012820 5870 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:56.984086 6550 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:11:56.997047 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 70782.8us
I20260929 14:11:56.998059 5342 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:11:56.999639 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:56.987746 5901 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:57.007167 5165 leader_election.cc:398] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3.
I20260929 14:11:57.018774 5165 leader_election.cc:401] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:11:57.019701 5165 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:57.019881 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:57.005381 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:56.993899 6675 raft_consensus.cc:2781] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:57.022291 7008 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.022686 6675 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:57.022898 7008 raft_consensus.cc:2726] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:57.023123 6675 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:11:57.023578 5271 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.231s user 0.005s sys 0.014s
W20260929 14:11:57.023871 5271 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.231s user 0.005s sys 0.014s
I20260929 14:11:56.994244 7045 raft_consensus.cc:2781] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:57.012403 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:56.996038 6984 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.016474 5199 raft_consensus.cc:1261] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:11:57.017601 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:57.027418 5276 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:57.020620 5273 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:57.028534 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:56.997248 6718 raft_consensus.cc:686] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:57.030858 6912 consensus_queue.cc:1046] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:11:57.030953 6718 consensus_queue.cc:237] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.032079 6912 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:57.024796 6550 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.033988 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:57.034610 5350 raft_consensus.cc:3037] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:57.035529 6550 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.035723 6735 consensus_queue.cc:1035] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:57.036069 6550 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.036567 6550 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.011418 6916 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 6
W20260929 14:11:57.039948 6998 log.cc:924] Time spent T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 1.031s user 0.005s sys 0.011s
I20260929 14:11:57.010459 6811 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.022204 6741 raft_consensus.cc:2726] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3.
I20260929 14:11:57.035732 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:57.012625 6909 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.029439 5276 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:57.013310 7028 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:57.044703 6984 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:57.012765 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:57.043179 5167 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:57.032584 6912 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.052086 7028 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:11:57.052177 6984 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.041491 6811 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.024660 6977 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.057161 5351 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:11:57.012867 6834 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.024863 7042 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.026141 7045 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:57.016099 7027 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.051566 5276 raft_consensus.cc:1226] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:57.069907 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:57.070600 5276 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:11:57.062501 5273 raft_consensus.cc:1261] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:57.049463 7039 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:57.072396 6984 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.065709 7042 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.073995 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.075564 6928 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.004895 6659 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.077560 6811 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:57.078358 6811 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.078876 6659 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:57.078811 6811 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.063053 6741 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:11:57.082347 6741 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:57.082345 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:57.083029 6741 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:57.083240 5901 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:11:57.085047 6912 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.090481 6732 raft_consensus.cc:3032] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:57.091245 6732 raft_consensus.cc:729] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:57.091800 7032 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:57.092670 6654 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.093883 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:57.066151 7034 log.cc:924] Time spent T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.173s user 0.001s sys 0.000s
I20260929 14:11:57.044384 5907 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:11:57.099997 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:57.113013 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 93381us
I20260929 14:11:57.114147 5165 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:57.117197 5165 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:57.119124 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:57.119433 5242 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:57.121798 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.123921 5907 raft_consensus.cc:2379] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 3: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:11:57.124418 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.122562 5342 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:57.126030 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.127928 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:57.134135 5350 raft_consensus.cc:2454] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:11:57.135507 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.135598 5870 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:57.136126 5342 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:57.044759 6845 consensus_queue.cc:1046] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
I20260929 14:11:57.068351 7045 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:57.146857 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:57.092352 6732 consensus_queue.cc:260] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.174420 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.125492 6659 raft_consensus.cc:2726] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:57.109370 6900 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.171960 5164 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:57.133466 6984 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:11:57.195472 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.132871 6909 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.142488 7008 raft_consensus.cc:2726] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:57.068079 7047 consensus_queue.cc:1035] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.155s
I20260929 14:11:57.144150 7029 raft_consensus.cc:2781] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:57.180394 5344 raft_consensus.cc:1226] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260929 14:11:57.209491 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.212666 7029 raft_consensus.cc:686] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:57.172266 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:11:57.220604 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 190944us
I20260929 14:11:57.200917 6732 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:57.179857 6928 consensus_queue.cc:1035] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.043s
I20260929 14:11:57.209059 6845 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.181833 6731 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.271502 6731 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:57.219302 7029 consensus_queue.cc:237] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.273113 5899 raft_consensus.cc:1261] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:57.217597 6984 raft_consensus.cc:686] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:57.224414 7008 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:11:57.225589 6811 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:57.245718 6845 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:57.294551 7051 log.cc:924] Time spent T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
W20260929 14:11:57.295024 6983 log.cc:924] Time spent T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 1.554s user 0.000s sys 0.006s
W20260929 14:11:57.298014 7050 log.cc:924] Time spent T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.240s user 0.001s sys 0.000s
I20260929 14:11:57.236174 6900 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.298727 5194 raft_consensus.cc:2354] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4.
I20260929 14:11:57.299674 5269 raft_consensus.cc:1261] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:11:57.300158 6946 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:57.303030 7042 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:57.308053 7044 log.cc:924] Time spent T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.207s user 0.000s sys 0.001s
I20260929 14:11:57.292420 5901 raft_consensus.cc:2379] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 4: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:11:57.300786 5268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 152) took 1575 ms. Trace:
I20260929 14:11:57.363197 5268 rpcz_store.cc:276] 0929 14:11:55.710521 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:55.717196 (+ 6675us) service_pool.cc:227] Handling call
0929 14:11:55.747486 (+ 30290us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:55.789767 (+ 42281us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:55.790585 (+ 818us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:55.797742 (+ 7157us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:11:55.801871 (+ 4129us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:57.236893 (+1435022us) spinlock_profiling.cc:243] Waited 29.8 ms on lock 0x7b54001090d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:57.241514 (+ 4621us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:57.282146 (+ 40632us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68514048}
I20260929 14:11:57.364187 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.365269 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:57.214733 6909 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.368265 6909 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.370211 6984 consensus_queue.cc:237] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.287269 7008 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:57.338207 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:57.373653 7008 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:11:57.373839 5269 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:57.347226 5903 raft_consensus.cc:2454] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:57.352106 5902 raft_consensus.cc:1226] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 2. Ops: []
I20260929 14:11:57.360317 6946 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.378696 6845 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.379379 6946 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:57.354085 6834 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.357564 5345 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 129) took 1270 ms. Trace:
I20260929 14:11:57.380533 5345 rpcz_store.cc:276] 0929 14:11:56.087381 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:56.087568 (+ 187us) service_pool.cc:227] Handling call
0929 14:11:56.088049 (+ 481us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:56.141161 (+ 53112us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:56.141208 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:56.735959 (+594751us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b2c000339d8. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda3450e11 00007fbda3460442 00007fbda4aa8c11 00007fbda4a6d58a 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772
0929 14:11:56.736460 (+ 501us) log.cc:842] Serialized 26 byte log entry
0929 14:11:56.932934 (+196474us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:11:57.142696 (+209762us) log.cc:842] Serialized 14 byte log entry
0929 14:11:57.145669 (+ 2973us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:57.275986 (+130317us) spinlock_profiling.cc:243] Waited 32.4 ms on lock 0x7b540027db54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:57.276042 (+ 56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:57.357136 (+ 81094us) raft_consensus.cc:1671] finished
0929 14:11:57.357166 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:57.357495 (+ 329us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12800,"prepare.queue_time_us":215610,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":49922,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":15408,"spinlock_wait_cycles":80216576,"thread_start_us":405267,"threads_started":2,"wal-append.queue_time_us":275270}
I20260929 14:11:57.301367 6550 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.373018 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:11:57.359128 7023 log.cc:924] Time spent T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.981s user 0.002s sys 0.007s
I20260929 14:11:57.399115 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1368) took 1291 ms. Trace:
I20260929 14:11:57.400709 5200 raft_consensus.cc:1226] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5. Ops: []
I20260929 14:11:57.399699 5193 rpcz_store.cc:276] 0929 14:11:56.107529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:56.109265 (+ 1736us) service_pool.cc:227] Handling call
0929 14:11:56.152743 (+ 43478us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:56.193403 (+ 40660us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:56.196215 (+ 2812us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:56.201171 (+ 4956us) log.cc:842] Serialized 26 byte log entry
0929 14:11:56.368574 (+167403us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:56.372736 (+ 4162us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:56.378166 (+ 5430us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:57.355467 (+977301us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda46efbee 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551 00007fbda46f8509 00007fbda46f8499
0929 14:11:57.398630 (+ 43163us) raft_consensus.cc:1671] finished
0929 14:11:57.398659 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:57.398968 (+ 309us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":924,"raft.queue_time_us":8236,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":11020,"spinlock_wait_cycles":3615616,"thread_start_us":145723,"threads_started":1,"wal-append.queue_time_us":131468}
I20260929 14:11:57.405465 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:57.395743 6741 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.409957 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.380736 6900 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.414104 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.421367 7058 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.421572 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.423281 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:57.422145 7058 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.442329 6916 raft_consensus.cc:2726] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6.
I20260929 14:11:57.444900 6675 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.448310 6978 ts_tablet_manager.cc:1918] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260929 14:11:57.400275 7028 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.366082 5268 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:57.472102 5198 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:11:57.479395 5199 raft_consensus.cc:2354] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5.
I20260929 14:11:57.392457 6989 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:57.473446 5314 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:57.468192 7016 log.cc:924] Time spent T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260929 14:11:57.486297 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.498656 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 157) took 1720 ms. Trace:
I20260929 14:11:57.499301 5271 rpcz_store.cc:276] 0929 14:11:55.778205 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:55.778375 (+ 170us) service_pool.cc:227] Handling call
0929 14:11:55.778599 (+ 224us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:55.778730 (+ 131us) raft_consensus.cc:1520] Early marking committed up to index 5
0929 14:11:57.024194 (+1245464us) log.cc:842] Serialized 14 byte log entry
0929 14:11:57.498187 (+473993us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:57.498262 (+ 75us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:11:57.498328 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:57.498421 (+ 93us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:57.498581 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":15397,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"thread_start_us":1704804,"threads_started":2}
I20260929 14:11:57.472358 7011 consensus_queue.cc:1035] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260929 14:11:57.474272 6946 consensus_queue.cc:1046] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260929 14:11:57.511637 7058 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.512542 6834 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:57.490959 6811 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:11:57.513598 6811 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:57.513527 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:57.514050 6811 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:57.514249 5193 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:57.515700 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 161) took 1038 ms. Trace:
I20260929 14:11:57.516018 5193 rpcz_store.cc:276] 0929 14:11:56.477666 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.514900 (+1037234us) service_pool.cc:227] Handling call
0929 14:11:57.515213 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:57.515374 (+ 161us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:57.515401 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:57.515445 (+ 44us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:11:57.515474 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:57.515532 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:57.515651 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:57.517549 7045 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.512804 6900 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:57.518865 6900 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.519830 7045 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.488997 7027 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.660948 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1376) took 1177 ms. Trace:
I20260929 14:11:57.660585 6909 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:57.583096 6675 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.665172 6909 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.665838 6741 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:57.661557 7028 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.580185 7008 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.671720 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.672190 5195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1357) took 1817 ms. Trace:
I20260929 14:11:57.673456 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:57.674211 5350 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 6
I20260929 14:11:57.525259 5314 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:57.674393 5316 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.505s user 0.003s sys 0.006s
W20260929 14:11:57.569960 6986 log.cc:924] Time spent T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
W20260929 14:11:57.680693 5316 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.511s user 0.003s sys 0.006s
I20260929 14:11:57.520195 5901 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:11:57.674219 6914 raft_consensus.cc:3032] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:11:57.684571 6935 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.684530 6900 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.492127 6978 log.cc:1196] T d653defbcc7e47ca89b35b1383e4e2c8 P c5ec116e39144881bdf1f2cd3e923cf5: Deleting WAL directory at /tmp/dist-test-task7GKXpD/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1790690998520575-4041-0/minicluster-data/ts-1-root/wals/d653defbcc7e47ca89b35b1383e4e2c8
I20260929 14:11:57.673732 5195 rpcz_store.cc:276] 0929 14:11:55.854553 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:55.877404 (+ 22851us) service_pool.cc:227] Handling call
0929 14:11:57.670637 (+1793233us) spinlock_profiling.cc:243] Waited 3.63 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:11:57.671764 (+ 1127us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:57.671917 (+ 153us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:57.671939 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:57.671976 (+ 37us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:11:57.672002 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:57.672047 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:57.672141 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8337792}
I20260929 14:11:57.555531 6946 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.512872 5268 raft_consensus.cc:1261] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:11:57.697463 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.661388 5193 rpcz_store.cc:276] 0929 14:11:56.483825 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.516943 (+1033118us) service_pool.cc:227] Handling call
0929 14:11:57.525337 (+ 8394us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:57.570757 (+ 45420us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:57.587468 (+ 16711us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:57.632358 (+ 44890us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:57.640922 (+ 8564us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:57.647588 (+ 6666us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:57.660886 (+ 13298us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:57.701773 6909 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:57.547646 7045 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:11:57.704885 5873 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.027s sys 0.027s
W20260929 14:11:57.729857 5873 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.580s user 0.027s sys 0.027s
W20260929 14:11:57.730161 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 628503us
W20260929 14:11:57.705446 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 180339us
W20260929 14:11:57.685947 6642 consensus_meta.cc:287] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.837s user 0.012s sys 0.001s
I20260929 14:11:57.726094 5195 raft_consensus.cc:1261] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260929 14:11:57.684361 6914 raft_consensus.cc:729] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:11:57.730926 5350 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 6.
I20260929 14:11:57.735733 5167 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
W20260929 14:11:57.735980 5164 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.536s user 0.012s sys 0.004s
W20260929 14:11:57.738998 5164 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.539s user 0.012s sys 0.004s
I20260929 14:11:57.737445 5195 pending_rounds.cc:76] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd: Aborting all ops after (but not including) 2
W20260929 14:11:57.739281 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 539085us
I20260929 14:11:57.737685 5316 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:57.491668 6989 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.739576 5195 pending_rounds.cc:97] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260929 14:11:57.741848 5870 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:57.744004 5195 raft_consensus.cc:2866] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:11:57.748157 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:57.748905 5903 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:57.675083 6834 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.760326 5903 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:11:57.736639 7052 raft_consensus.cc:2781] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:57.740633 7045 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.704198 6845 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260929 14:11:57.741904 6822 consensus_queue.cc:1035] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.026s
I20260929 14:11:57.753796 6741 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.724489 6946 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.744310 6732 raft_consensus.cc:2781] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:57.761868 6907 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:57.764431 7036 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:11:57.767959 6916 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.765127 7058 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.764719 6914 consensus_queue.cc:260] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.765604 6675 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.761605 6562 raft_consensus.cc:2741] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260929 14:11:57.768409 5195 raft_consensus.cc:1261] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
W20260929 14:11:57.751101 6920 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s user 0.011s sys 0.015s
I20260929 14:11:57.769055 5167 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:57.693073 6935 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.766691 7052 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:57.774724 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:57.743021 6642 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.747231 7035 consensus_queue.cc:1046] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260929 14:11:57.744094 7032 consensus_queue.cc:1035] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260929 14:11:57.789117 5195 pending_rounds.cc:76] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd: Aborting all ops after (but not including) 1
I20260929 14:11:57.801492 5195 pending_rounds.cc:97] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260929 14:11:57.801819 5195 raft_consensus.cc:2866] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:11:57.779484 6921 consensus_queue.cc:1035] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.019s
I20260929 14:11:57.802999 6978 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50768 (request call id 908) took 2353 ms. Trace:
I20260929 14:11:57.803267 5899 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:57.803437 6978 rpcz_store.cc:276] 0929 14:11:55.449645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:55.450779 (+ 1134us) service_pool.cc:227] Handling call
0929 14:11:55.704495 (+253716us) ts_tablet_manager.cc:1145] Deleting tablet d653defbcc7e47ca89b35b1383e4e2c8
0929 14:11:55.733377 (+ 28882us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0929 14:11:55.910907 (+177530us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b5400133154. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a75fe9 00007fbda99fdb72 00007fbda9a2de10 00007fbda9a14e0d 00007fbda9a14db9 00007fbda9a14d49 00007fbda9a14d11 00007fbda9a13f3d 00007fbdaada9204 00007fbdaada9039
0929 14:11:57.089287 (+1178380us) tablet_metadata.cc:641] Metadata flushed
0929 14:11:57.447974 (+358687us) tablet_metadata.cc:641] Metadata flushed
0929 14:11:57.802924 (+354950us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5970,"spinlock_wait_cycles":361951488,"tablet-delete.queue_time_us":206766,"thread_start_us":150857,"threads_started":1}
I20260929 14:11:57.785557 6916 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.806892 6916 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:57.808859 7035 raft_consensus.cc:3032] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:57.809378 7035 raft_consensus.cc:729] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:57.810374 7035 consensus_queue.cc:260] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.811911 7035 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:11:57.790151 5165 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.663s user 0.011s sys 0.059s
I20260929 14:11:57.772915 6550 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:57.813576 6550 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.800861 7052 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:57.815560 6946 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.816852 5343 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47252 (request call id 1766) took 1513 ms. Trace:
I20260929 14:11:57.817243 5343 rpcz_store.cc:276] 0929 14:11:56.303671 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:56.306997 (+ 3326us) service_pool.cc:227] Handling call
0929 14:11:57.816665 (+1509668us) spinlock_profiling.cc:243] Waited 76.7 ms on lock 0x7b540012f554. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:57.816772 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":176405760}
I20260929 14:11:57.817775 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.786586 7058 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.818753 6642 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.819636 5197 raft_consensus.cc:2354] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:11:57.819962 7058 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.785362 7036 raft_consensus.cc:686] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 6 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:57.793123 6914 raft_consensus.cc:3037] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:11:57.822002 7036 consensus_queue.cc:237] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.788125 7000 consensus_queue.cc:1035] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.003s
I20260929 14:11:57.821060 5197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52188 (request call id 1385) took 1555 ms. Trace:
I20260929 14:11:57.875991 5197 rpcz_store.cc:276] 0929 14:11:56.265336 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:56.894065 (+628729us) service_pool.cc:227] Handling call
0929 14:11:57.820832 (+926767us) spinlock_profiling.cc:243] Waited 924 ms on lock 0x7b5400408b54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:57.820985 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2126163200}
W20260929 14:11:57.823851 5313 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.568s user 0.001s sys 0.016s
W20260929 14:11:57.877486 5313 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.706s user 0.001s sys 0.016s
I20260929 14:11:57.788019 7033 consensus_queue.cc:1035] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.023s
W20260929 14:11:57.836548 5165 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.710s user 0.011s sys 0.059s
I20260929 14:11:57.770546 6811 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.786895 6732 raft_consensus.cc:686] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:57.824584 5084 catalog_manager.cc:4841] TS c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): tablet d653defbcc7e47ca89b35b1383e4e2c8 (table test-workload [id=7efbef67b40f466dae10d9b6ed991948]) successfully deleted
W20260929 14:11:57.881116 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 754416us
W20260929 14:11:57.881572 5084 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 57018us
I20260929 14:11:57.815567 6550 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.848098 5899 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:57.852540 6935 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:57.791867 6675 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.854775 5200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1382) took 1686 ms. Trace:
I20260929 14:11:57.903446 5200 rpcz_store.cc:276] 0929 14:11:56.167977 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:56.990216 (+822239us) service_pool.cc:227] Handling call
0929 14:11:57.400420 (+410204us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:57.401360 (+ 940us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:57.854528 (+453168us) spinlock_profiling.cc:243] Waited 410 ms on lock 0x7b54003db650. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6ca09 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:57.854703 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":942729088}
I20260929 14:11:57.782670 6907 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.864079 5194 raft_consensus.cc:1226] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260929 14:11:57.782378 6741 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.866596 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.821141 6989 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:57.792023 6920 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s user 0.012s sys 0.015s
I20260929 14:11:57.879164 6642 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.934342 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:57.934381 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.935204 5350 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 4
I20260929 14:11:57.910684 6550 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.938376 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:57.880159 6659 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:11:57.936820 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.936978 5167 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:57.925030 6823 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.940584 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:57.931308 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:57.942641 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:57.944306 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.966498 5903 raft_consensus.cc:2454] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:57.945937 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:57.967315 5344 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:57.928774 6822 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.968277 5350 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:57.956593 5904 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45248 (request call id 278) took 1472 ms. Trace:
I20260929 14:11:57.962990 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:57.932392 7003 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:57.958899 6907 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:57.968432 5347 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 5
I20260929 14:11:57.950079 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:57.884117 7058 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:57.987706 5346 raft_consensus.cc:2379] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 6: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:11:57.928228 6642 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:57.972781 7029 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.697s user 0.010s sys 0.043s
W20260929 14:11:57.965653 5192 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s user 0.016s sys 0.065s
I20260929 14:11:57.943490 6550 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.964190 7008 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:57.997426 5192 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s user 0.017s sys 0.065s
I20260929 14:11:57.951391 6916 consensus_queue.cc:1035] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
W20260929 14:11:57.995952 6718 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.756s user 0.013s sys 0.029s
W20260929 14:11:57.991082 6735 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.670s user 0.002s sys 0.031s
W20260929 14:11:58.004586 5907 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.769s user 0.012s sys 0.029s
W20260929 14:11:58.021423 5270 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.705s user 0.026s sys 0.034s
I20260929 14:11:57.992154 6587 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:11:57.995373 7029 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.719s user 0.011s sys 0.043s
W20260929 14:11:58.037757 5273 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.022s sys 0.068s
W20260929 14:11:58.039505 5907 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.804s user 0.012s sys 0.029s
W20260929 14:11:58.039613 5270 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.723s user 0.026s sys 0.034s
W20260929 14:11:58.044934 5273 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.744s user 0.022s sys 0.068s
I20260929 14:11:57.977425 6822 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:57.959853 6732 consensus_queue.cc:237] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:57.974596 5904 rpcz_store.cc:276] 0929 14:11:56.484662 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:56.485974 (+ 1312us) service_pool.cc:227] Handling call
0929 14:11:57.956356 (+1470382us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b540029a254. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:57.956490 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":242291584}
I20260929 14:11:58.044485 7065 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:58.069329 7065 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.071962 7065 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:58.038928 6718 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.801s user 0.013s sys 0.029s
W20260929 14:11:58.039389 6735 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.719s user 0.002s sys 0.032s
I20260929 14:11:58.020347 7011 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:58.073511 7011 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.075948 7011 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:57.976971 6823 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.078166 6823 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:57.989022 7003 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.080348 7003 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:58.081552 5342 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
W20260929 14:11:58.082505 5274 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.676s user 0.007s sys 0.019s
I20260929 14:11:58.015141 7045 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:58.004338 7058 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.138612 5351 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1471) took 1144 ms. Trace:
I20260929 14:11:58.041975 7008 raft_consensus.cc:2781] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election won for term 4
W20260929 14:11:58.116741 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 345494us
W20260929 14:11:58.133966 5198 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52188 (request call id 1371) took 1976 ms (client timeout 1723 ms). Trace:
W20260929 14:11:58.184440 5198 rpcz_store.cc:269] 0929 14:11:56.157451 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:56.157674 (+ 223us) service_pool.cc:227] Handling call
0929 14:11:57.035590 (+877916us) spinlock_profiling.cc:243] Waited 1.65 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:11:58.133791 (+1098201us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b5400012554. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:58.133902 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":978064640}
I20260929 14:11:58.138067 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52182 (request call id 1381) took 1013 ms. Trace:
I20260929 14:11:58.136286 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:58.168486 5351 rpcz_store.cc:276] 0929 14:11:56.994302 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.005098 (+ 10796us) service_pool.cc:227] Handling call
0929 14:11:58.138457 (+1133359us) spinlock_profiling.cc:243] Waited 36.4 ms on lock 0x7b540012d254. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:58.138560 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":83748224}
I20260929 14:11:58.187829 5239 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:58.185853 5199 rpcz_store.cc:276] 0929 14:11:57.124301 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.127620 (+ 3319us) service_pool.cc:227] Handling call
0929 14:11:58.137885 (+1010265us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b54002cc754. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:58.138012 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":795970944}
W20260929 14:11:58.082866 5274 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.677s user 0.007s sys 0.019s
W20260929 14:11:58.083377 6941 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.986s user 0.022s sys 0.048s
I20260929 14:11:58.193459 5906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45248 (request call id 283) took 1041 ms. Trace:
W20260929 14:11:58.128223 7067 log.cc:924] Time spent T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
W20260929 14:11:58.214267 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 321213us
I20260929 14:11:58.140184 5898 raft_consensus.cc:1226] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: [2.5-3.6]
W20260929 14:11:58.232765 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 44208.3us
I20260929 14:11:58.103181 6675 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:58.128036 6822 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:58.244349 5906 rpcz_store.cc:276] 0929 14:11:57.152121 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.152671 (+ 550us) service_pool.cc:227] Handling call
0929 14:11:58.136131 (+983460us) spinlock_profiling.cc:243] Waited 19.8 ms on lock 0x7b5800027698. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:11:58.193344 (+ 57213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":45528832}
I20260929 14:11:58.282449 5316 leader_election.cc:398] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4.
I20260929 14:11:58.282874 5316 leader_election.cc:401] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:11:58.130448 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1379) took 1022 ms. Trace:
I20260929 14:11:58.108327 6811 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:11:58.191076 5348 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.500s user 0.028s sys 0.043s
I20260929 14:11:58.284613 5165 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:58.121440 6741 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:58.273703 6941 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.147s user 0.037s sys 0.058s
W20260929 14:11:58.290582 6717 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.629s user 0.005s sys 0.017s
I20260929 14:11:58.212741 7008 raft_consensus.cc:686] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:58.268325 6984 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:11:58.293325 7031 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 1.525s user 0.010s sys 0.019s
W20260929 14:11:58.285842 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 348916us
I20260929 14:11:58.233062 6914 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:58.234071 6946 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:11:58.294636 6845 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:58.284317 7002 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:58.331522 5316 leader_election.cc:398] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5.
I20260929 14:11:58.325788 5344 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:58.291676 6822 raft_consensus.cc:2726] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4.
I20260929 14:11:58.345355 6946 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:58.282987 6987 consensus_queue.cc:1035] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260929 14:11:58.382242 6946 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:11:58.383082 6999 log.cc:924] Time spent T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 2.378s user 0.000s sys 0.015s
I20260929 14:11:58.313329 6675 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260929 14:11:58.384543 7077 log.cc:924] Time spent T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20260929 14:11:58.384773 6675 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:58.385176 6675 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:11:58.386022 7076 log.cc:924] Time spent T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260929 14:11:58.294703 5192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1369) took 2169 ms. Trace:
W20260929 14:11:58.306798 5271 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.716s user 0.007s sys 0.029s
W20260929 14:11:58.390803 5271 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.819s user 0.009s sys 0.029s
W20260929 14:11:58.345643 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 428265us
W20260929 14:11:58.325027 7052 consensus_meta.cc:287] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.510s user 0.006s sys 0.005s
I20260929 14:11:58.363598 5316 leader_election.cc:401] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:11:58.364723 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:58.395197 5275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1357) took 2552 ms. Trace:
I20260929 14:11:58.322340 5898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45248 (request call id 276) took 1848 ms. Trace:
I20260929 14:11:58.288545 7003 consensus_queue.cc:1035] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.046s
I20260929 14:11:58.331007 6642 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:58.329873 6562 consensus_queue.cc:1046] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
W20260929 14:11:58.323271 6717 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.659s user 0.005s sys 0.017s
W20260929 14:11:58.343487 7031 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 1.567s user 0.011s sys 0.020s
I20260929 14:11:58.333962 5167 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:58.440819 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:58.382984 7058 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:58.325302 5274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1363) took 1981 ms. Trace:
I20260929 14:11:58.387714 5270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1379) took 1100 ms. Trace:
I20260929 14:11:58.390138 5347 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:11:58.395659 5275 rpcz_store.cc:276] 0929 14:11:55.843187 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:55.850202 (+ 7015us) service_pool.cc:227] Handling call
0929 14:11:55.850571 (+ 369us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:55.858327 (+ 7756us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:55.858373 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:55.858882 (+ 509us) log.cc:842] Serialized 26 byte log entry
0929 14:11:55.940372 (+ 81490us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:11:55.940420 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:55.940479 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:58.382523 (+2442044us) spinlock_profiling.cc:243] Waited 31.3 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda46efbee 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551 00007fbda46f8509 00007fbda46f8499
0929 14:11:58.387269 (+ 4746us) raft_consensus.cc:1671] finished
0929 14:11:58.387290 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:58.395122 (+ 7832us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7323,"raft.queue_time_us":10579,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109,"spinlock_wait_cycles":71987968,"thread_start_us":19340,"threads_started":1,"wal-append.queue_time_us":86976}
I20260929 14:11:58.399420 5164 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:58.289341 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 357403us
W20260929 14:11:58.414553 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 83222.9us
I20260929 14:11:58.330951 6984 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.287614 5193 rpcz_store.cc:276] 0929 14:11:57.108116 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.739250 (+631134us) service_pool.cc:227] Handling call
0929 14:11:57.767934 (+ 28684us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:57.938730 (+170796us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:57.940863 (+ 2133us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:57.944179 (+ 3316us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:57.952787 (+ 8608us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:58.130203 (+177416us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b540031ef54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:58.130246 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:58.130384 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":370120576}
I20260929 14:11:58.363533 6914 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.390632 5192 rpcz_store.cc:276] 0929 14:11:56.122661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:56.135863 (+ 13202us) service_pool.cc:227] Handling call
0929 14:11:57.094373 (+958510us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:11:57.095285 (+ 912us) raft_consensus.cc:1457] Updating replica for 2 ops
0929 14:11:57.095798 (+ 513us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:57.095837 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0929 14:11:57.214526 (+118689us) log.cc:842] Serialized 52 byte log entry
0929 14:11:58.040896 (+826370us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:58.119611 (+ 78715us) log.cc:842] Serialized 14 byte log entry
0929 14:11:58.155474 (+ 35863us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:58.189138 (+ 33664us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:58.192404 (+ 3266us) raft_consensus.cc:1671] finished
0929 14:11:58.268971 (+ 76567us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:58.291849 (+ 22878us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":778,"prepare.queue_time_us":43388,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":28136,"raft.run_cpu_time_us":365,"raft.run_wall_time_us":24669,"spinlock_wait_cycles":5997312,"thread_start_us":730434,"threads_started":1,"wal-append.queue_time_us":883519}
I20260929 14:11:58.396977 7052 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.416401 6935 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:58.414585 7008 consensus_queue.cc:237] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:58.396916 6900 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:58.415299 6659 raft_consensus.cc:686] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:58.545285 6935 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:58.546144 6659 consensus_queue.cc:237] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:58.548493 6935 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:58.395632 5348 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 131) took 1432 ms. Trace:
I20260929 14:11:58.549772 6900 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:58.549629 5348 rpcz_store.cc:276] 0929 14:11:56.963107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:56.968361 (+ 5254us) service_pool.cc:227] Handling call
0929 14:11:56.993862 (+ 25501us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:56.994068 (+ 206us) raft_consensus.cc:1520] Early marking committed up to index 5
0929 14:11:57.490977 (+496909us) log.cc:842] Serialized 14 byte log entry
0929 14:11:58.342116 (+851139us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:58.348813 (+ 6697us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:11:58.363821 (+ 15008us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:58.376615 (+ 12794us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:58.388685 (+ 12070us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7713,"prepare.queue_time_us":154793,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":10490,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":6388,"thread_start_us":1091482,"threads_started":2}
I20260929 14:11:58.551062 6984 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:58.446058 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 112201us
I20260929 14:11:58.551993 5167 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:58.464605 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 170928us
W20260929 14:11:58.465574 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 66187.4us
I20260929 14:11:58.554049 6914 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:58.554266 7073 log.cc:924] Time spent T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.166s user 0.000s sys 0.000s
I20260929 14:11:58.473067 5196 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:58.447264 5270 rpcz_store.cc:276] 0929 14:11:57.287877 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.288525 (+ 648us) service_pool.cc:227] Handling call
0929 14:11:57.288969 (+ 444us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:57.315464 (+ 26495us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:57.315528 (+ 64us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:57.316084 (+ 556us) log.cc:842] Serialized 26 byte log entry
0929 14:11:58.043979 (+727895us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:58.044033 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:58.044207 (+ 174us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:58.387492 (+343285us) raft_consensus.cc:1671] finished
0929 14:11:58.387510 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:58.387655 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25942,"raft.queue_time_us":27672,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":5209,"thread_start_us":727373,"threads_started":1,"wal-append.queue_time_us":952581}
W20260929 14:11:58.554939 5196 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52188 (request call id 1372) took 2397 ms (client timeout 1720 ms). Trace:
W20260929 14:11:58.555290 5196 rpcz_store.cc:269] 0929 14:11:56.157768 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:56.157876 (+ 108us) service_pool.cc:227] Handling call
0929 14:11:58.440649 (+2282773us) spinlock_profiling.cc:243] Waited 1.61 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:11:58.554874 (+114225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3701504}
I20260929 14:11:58.556388 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:58.556566 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:58.429569 7079 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:58.465574 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:58.612505 5199 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:58.613021 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:58.613745 5270 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:11:58.613763 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:11:58.505468 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 57083.7us
I20260929 14:11:58.614720 5199 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:58.615567 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:58.616565 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 64629.8us
I20260929 14:11:58.466709 5198 raft_consensus.cc:1226] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:11:58.378712 6845 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.487233 5316 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:58.429562 7007 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:58.536672 5906 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:11:58.415787 7036 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:58.451560 7083 raft_consensus.cc:2726] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5.
I20260929 14:11:58.612531 5196 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:11:58.617941 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:58.710179 5198 raft_consensus.cc:2396] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260929 14:11:58.710800 5201 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s user 0.047s sys 0.056s
I20260929 14:11:58.710817 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:58.711166 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:11:58.711613 5201 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.580s user 0.047s sys 0.056s
I20260929 14:11:58.711717 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:58.713073 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:58.713128 5275 raft_consensus.cc:2454] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:11:58.715142 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:58.377199 7002 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.618801 7089 raft_consensus.cc:2726] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260929 14:11:58.619988 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:58.621855 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:58.698927 5873 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.478s (SENT) [suppressed 42 similar messages]
I20260929 14:11:58.448927 6642 raft_consensus.cc:2781] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:58.397652 5898 rpcz_store.cc:276] 0929 14:11:56.473885 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:56.476138 (+ 2253us) service_pool.cc:227] Handling call
0929 14:11:58.139753 (+1663615us) spinlock_profiling.cc:243] Waited 7.1 ms on lock 0x7b5800027698. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0dbce 000000000034aea9 00007fbda9939a30 00007fbda99ff7b4 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:11:58.139911 (+ 158us) raft_consensus.cc:1457] Updating replica for 2 ops
0929 14:11:58.285471 (+145560us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:58.319685 (+ 34214us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16331136}
I20260929 14:11:58.736744 6642 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 14:11:58.736943 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 249844us
I20260929 14:11:58.737257 6642 raft_consensus.cc:3037] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:58.737740 6917 raft_consensus.cc:2741] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260929 14:11:58.737740 5316 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:58.709170 6920 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:58.740610 6920 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:58.620954 7092 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:58.724604 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:58.741492 7092 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:58.706522 7035 consensus_meta.cc:287] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.894s user 0.013s sys 0.001s
I20260929 14:11:58.567288 6867 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:58.751222 6867 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:58.752388 7089 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:58.752941 7089 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.717553 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:58.748412 7003 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:11:58.755509 6867 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:58.497754 7058 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:58.756486 7058 raft_consensus.cc:686] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
W20260929 14:11:58.758074 6587 consensus_meta.cc:287] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.715s user 0.010s sys 0.000s
I20260929 14:11:58.758672 5272 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:58.457535 6717 raft_consensus.cc:3032] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Stepping down as leader of term 2
W20260929 14:11:58.710222 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 85959.1us
I20260929 14:11:58.716998 5871 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:58.759511 6912 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:11:58.763805 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 46940.3us
I20260929 14:11:58.722520 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:58.446576 5274 rpcz_store.cc:276] 0929 14:11:56.341675 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:56.343812 (+ 2137us) service_pool.cc:227] Handling call
0929 14:11:56.375558 (+ 31746us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:56.436778 (+ 61220us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:11:56.485846 (+ 49068us) log.cc:842] Serialized 14 byte log entry
0929 14:11:57.405804 (+919958us) spinlock_profiling.cc:243] Waited 84.1 ms on lock 0x7b180032f440. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b27345 00007fbdaa95ea39 00007fbda1b448bb 00007fbda1b42cbf 00007fbda1b42155 00007fbda4a653d0 00007fbda4a77576 00007fbda4a880d7 00007fbda4a88058
0929 14:11:58.264897 (+859093us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:58.283938 (+ 19041us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:11:58.290068 (+ 6130us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:58.295849 (+ 5781us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:58.322629 (+ 26780us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":33912,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":4635,"raft.queue_time_us":157994,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":6805,"spinlock_wait_cycles":193338240,"thread_start_us":790490,"threads_started":1}
I20260929 14:11:58.710695 7079 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.727911 6675 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.707044 6822 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:58.730305 7068 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:58.730758 7002 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:58.718780 7036 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:58.734419 5314 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:58.734558 5270 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:11:58.802649 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:58.763006 6550 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:58.763257 6717 raft_consensus.cc:729] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:58.735656 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:58.567513 6900 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:58.702373 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:58.720407 7007 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:58.751446 6917 consensus_queue.cc:1046] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260929 14:11:58.720837 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 167946us
I20260929 14:11:58.770668 6587 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:58.735316 7065 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:58.764127 6912 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.757696 7058 consensus_queue.cc:237] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.744290 7087 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:58.748472 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:58.782267 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:58.776649 5199 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
W20260929 14:11:58.801591 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 67239.8us
I20260929 14:11:58.814375 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:58.768806 7029 raft_consensus.cc:2781] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:11:58.741925 5193 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:58.779733 6984 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:58.810513 7003 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:11:58.800448 6914 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:58.803364 5270 raft_consensus.cc:2454] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:58.936687 5314 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:58.937144 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:58.944186 5351 raft_consensus.cc:2454] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:58.802052 6822 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:58.803668 7068 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.948987 6822 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:58.950160 6946 consensus_meta.cc:287] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.567s user 0.010s sys 0.000s
I20260929 14:11:58.950631 7068 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:58.950600 6946 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.923280 7094 consensus_queue.cc:1035] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.119s
I20260929 14:11:58.952806 6946 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:58.820628 6587 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.954725 7052 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:58.943692 7003 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:58.955667 7003 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:11:58.958194 5274 raft_consensus.cc:2379] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 5: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:11:58.959940 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:58.960567 5274 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:58.946988 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:58.947363 7081 log.cc:924] Time spent T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20260929 14:11:58.937570 7004 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:58.931689 7089 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:58.929373 6550 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:58.963001 5164 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:58.963332 5274 raft_consensus.cc:1261] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:11:58.975698 5270 raft_consensus.cc:1261] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:11:58.948146 5314 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:58.976506 5270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1389) took 1613 ms. Trace:
I20260929 14:11:58.976967 5270 rpcz_store.cc:276] 0929 14:11:57.363188 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:58.973648 (+1610460us) service_pool.cc:227] Handling call
0929 14:11:58.973930 (+ 282us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:58.976208 (+ 2278us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:58.976428 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:58.939491 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:58.955173 6587 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:58.939859 7029 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:58.922333 7065 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:58.978188 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:58.976194 5274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1388) took 1613 ms. Trace:
I20260929 14:11:58.985859 5274 rpcz_store.cc:276] 0929 14:11:57.362843 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:58.961164 (+1598321us) service_pool.cc:227] Handling call
0929 14:11:58.963044 (+ 1880us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:58.975917 (+ 12873us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:58.976127 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:58.986714 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:58.987435 5270 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:58.980276 5314 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:58.970512 6917 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:11:58.949265 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:58.990778 5274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1390) took 1627 ms. Trace:
I20260929 14:11:58.991240 5274 rpcz_store.cc:276] 0929 14:11:57.363487 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:58.986781 (+1623294us) service_pool.cc:227] Handling call
0929 14:11:58.987075 (+ 294us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:58.987266 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:58.987290 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:58.987325 (+ 35us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:58.990503 (+ 3178us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:58.990584 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:58.990715 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:58.938074 7087 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:58.988917 6917 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:58.996140 6917 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:11:58.997187 7087 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:58.998034 7092 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.001977 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.005391 7079 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.006183 6920 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:58.984902 7029 raft_consensus.cc:3037] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:59.008184 6917 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.009330 5351 raft_consensus.cc:1226] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6. Ops: []
I20260929 14:11:59.011195 6917 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:58.974371 7004 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:58.981323 6987 raft_consensus.cc:2781] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:11:59.013000 6987 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:58.945760 6984 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:58.979390 6550 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.015174 6984 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:58.937579 6717 consensus_queue.cc:260] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.015983 6550 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:58.979329 7089 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:59.016472 6845 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.016856 7089 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.017202 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.018390 6642 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:59.018898 7089 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.019812 6675 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.020398 6642 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:59.021320 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.022126 7065 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:59.022871 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.020231 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.024174 5348 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:59.013360 6987 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:58.984269 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.023583 5193 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:59.027233 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1384) took 1666 ms. Trace:
I20260929 14:11:58.969280 6914 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.028118 5164 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:59.024308 5342 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47252 (request call id 1793) took 1077 ms. Trace:
I20260929 14:11:59.027606 5271 rpcz_store.cc:276] 0929 14:11:57.361385 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.546222 (+184837us) service_pool.cc:227] Handling call
0929 14:11:57.570492 (+ 24270us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:57.570921 (+ 429us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:57.570964 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:57.571398 (+ 434us) log.cc:842] Serialized 26 byte log entry
0929 14:11:58.391217 (+819819us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:58.391270 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:58.391356 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:59.027015 (+635659us) raft_consensus.cc:1671] finished
0929 14:11:59.027040 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.027168 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":11407,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":819500,"threads_started":1,"wal-append.queue_time_us":1198503}
W20260929 14:11:59.029753 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 49560.1us
I20260929 14:11:59.030946 5342 rpcz_store.cc:276] 0929 14:11:57.946736 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.947314 (+ 578us) service_pool.cc:227] Handling call
0929 14:11:59.024135 (+1076821us) spinlock_profiling.cc:243] Waited 112 ms on lock 0x7b54002530d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:59.024243 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":258261376}
I20260929 14:11:59.035377 7065 raft_consensus.cc:2781] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:59.035812 7065 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:59.036911 5164 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:59.042477 6583 consensus_queue.cc:1035] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:59.042647 6493 consensus_queue.cc:1035] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:11:58.988092 7045 raft_consensus.cc:2781] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:11:59.036114 7065 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:59.004786 7007 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:59.056075 5903 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.034s sys 0.018s
I20260929 14:11:59.044456 6642 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:59.026440 7029 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:59.056517 5903 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s user 0.034s sys 0.018s
I20260929 14:11:59.056938 5270 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:59.036319 6917 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:59.058841 5314 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:59.060341 7004 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:59.044337 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.062536 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:59.048990 7045 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:59.064344 5199 raft_consensus.cc:2354] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:11:59.063119 5270 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
W20260929 14:11:59.072315 5242 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.606s user 0.027s sys 0.052s
W20260929 14:11:59.072718 5242 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.607s user 0.027s sys 0.052s
W20260929 14:11:59.082618 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 383680us
I20260929 14:11:59.057463 6642 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:59.057907 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:59.066681 7095 log.cc:924] Time spent T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20260929 14:11:59.095986 5270 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:11:59.097467 5314 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:59.103629 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.104288 5270 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:11:59.105293 5314 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:59.058414 6917 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:11:59.070044 7004 raft_consensus.cc:2781] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:59.066583 7045 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:59.069562 6675 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:59.123574 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:59.078930 6920 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:59.083765 7042 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:59.100162 6642 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:59.119339 6403 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:11:59.122318 6917 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:59.125231 6751 raft_consensus.cc:2781] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260929 14:11:59.149073 5165 connection.cc:232] RPC call timeout handler was delayed by -2.111826673999758s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19390000000000002s for a response
I20260929 14:11:59.151638 5906 raft_consensus.cc:1261] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260929 14:11:59.155265 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 413363us
I20260929 14:11:59.124653 6587 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:59.128358 7036 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:59.123490 5193 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:59.084988 7003 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.165310 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.174460 5197 raft_consensus.cc:2396] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260929 14:11:59.171254 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.133415 7004 raft_consensus.cc:686] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:59.175969 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.177260 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:59.177775 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.179715 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.180502 5197 raft_consensus.cc:2454] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:59.135829 6984 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:11:59.185179 5242 leader_election.cc:334] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.551s (SENT)
I20260929 14:11:59.185811 5242 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:59.154623 6845 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:59.152637 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:59.156744 6751 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:59.197197 6751 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:11:59.199361 5316 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:59.203926 5201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1371) took 3080 ms. Trace:
I20260929 14:11:59.203881 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:11:59.204797 5906 raft_consensus.cc:3037] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:59.206640 5316 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:11:59.207473 5242 leader_election.cc:334] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.553s (SENT)
I20260929 14:11:59.143946 6675 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.229300 6675 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.188828 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:59.169826 6587 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.144023 6920 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:59.200203 7058 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:59.211859 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:59.213557 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:59.178540 6928 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:59.215801 5902 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45232 (request call id 332) took 1474 ms. Trace:
I20260929 14:11:59.217516 7007 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:11:59.201511 5900 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45248 (request call id 279) took 2700 ms (client timeout 1844 ms). Trace:
I20260929 14:11:59.218024 7002 raft_consensus.cc:3032] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:11:59.218158 7004 consensus_queue.cc:237] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.226107 6583 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:59.208220 5242 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:59.239485 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.233142 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:59.234892 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.204490 5201 rpcz_store.cc:276] 0929 14:11:56.123370 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:56.123484 (+ 114us) service_pool.cc:227] Handling call
0929 14:11:57.924438 (+1800954us) spinlock_profiling.cc:243] Waited 2.65 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:11:57.924717 (+ 279us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:58.109150 (+184433us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:11:58.109198 (+ 48us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:58.129094 (+ 19896us) log.cc:842] Serialized 26 byte log entry
0929 14:11:58.712058 (+582964us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:11:58.712099 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:58.712174 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:59.196754 (+484580us) raft_consensus.cc:1671] finished
0929 14:11:59.196774 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.203856 (+ 7082us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16951,"raft.queue_time_us":67490,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":6763,"spinlock_wait_cycles":6089472,"thread_start_us":764477,"threads_started":2,"wal-append.queue_time_us":688146}
I20260929 14:11:59.183043 7079 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:59.183257 6403 raft_consensus.cc:686] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:59.170162 7087 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260929 14:11:59.155782 7042 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:59.279261 7008 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:59.253022 6922 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:11:59.232163 7097 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:59.271548 6493 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:59.326009 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.284940 7003 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:59.206660 6845 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:59.206013 6984 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:59.287181 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:59.282527 5900 rpcz_store.cc:269] 0929 14:11:56.492037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:56.498178 (+ 6141us) service_pool.cc:227] Handling call
0929 14:11:59.173486 (+2675308us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54007a1554. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:11:59.185503 (+ 12017us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":579028352}
I20260929 14:11:59.287949 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:59.288550 7029 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:11:59.288911 7098 log.cc:924] Time spent T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.123s user 0.000s sys 0.000s
W20260929 14:11:59.354308 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 113155us
I20260929 14:11:59.253950 6914 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.289772 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:59.280526 6928 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:59.309520 7079 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.263718 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:59.301791 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 143269us
I20260929 14:11:59.322325 7008 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:59.232366 7101 log.cc:924] Time spent T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260929 14:11:59.232024 7040 consensus_queue.cc:1046] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.061s
I20260929 14:11:59.301105 5197 raft_consensus.cc:2362] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2. Re-sending same reply.
W20260929 14:11:59.322942 5165 leader_election.cc:334] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.939s (SENT)
I20260929 14:11:59.384963 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:59.385279 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 62375.1us
I20260929 14:11:59.385600 5197 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
W20260929 14:11:59.301659 7074 log.cc:924] Time spent T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.914s user 0.004s sys 0.000s
I20260929 14:11:59.305438 6583 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:59.324244 6403 consensus_queue.cc:237] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.292172 7002 raft_consensus.cc:729] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:59.296422 7109 raft_consensus.cc:2726] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:11:59.329149 5201 raft_consensus.cc:2379] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 5: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:59.301275 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:59.328193 7097 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:59.313901 6587 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.342093 5274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1391) took 1978 ms. Trace:
I20260929 14:11:59.346055 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.326519 6493 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 14:11:59.352331 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 144892us
W20260929 14:11:59.414794 5242 leader_election.cc:334] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.565s (SENT)
I20260929 14:11:59.330024 6922 raft_consensus.cc:686] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:59.370561 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.332459 7049 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:59.280963 7058 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.283584 5902 rpcz_store.cc:276] 0929 14:11:57.740985 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.742564 (+ 1579us) service_pool.cc:227] Handling call
0929 14:11:59.214848 (+1472284us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x7b5800027698. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:11:59.215107 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:59.215352 (+ 245us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:59.215392 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:59.215456 (+ 64us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:11:59.215516 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:59.215592 (+ 76us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.215726 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6516992}
I20260929 14:11:59.304682 6834 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:59.379477 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.426373 5350 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:59.386418 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.350466 5194 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:11:59.384536 5275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1386) took 2022 ms. Trace:
I20260929 14:11:59.387902 5167 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:59.390018 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:59.391264 5898 raft_consensus.cc:1261] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 14:11:59.391712 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.394104 5267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1369) took 3016 ms. Trace:
I20260929 14:11:59.394354 5316 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:59.372854 6914 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:59.333974 6987 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.396988 6920 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:11:59.400715 7113 log.cc:924] Time spent T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260929 14:11:59.402601 6928 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:59.403669 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.403940 7042 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:59.404752 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.330519 5906 raft_consensus.cc:2454] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:59.402637 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.409403 5274 rpcz_store.cc:276] 0929 14:11:57.363779 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:58.994150 (+1630371us) service_pool.cc:227] Handling call
0929 14:11:58.994455 (+ 305us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:58.994762 (+ 307us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:58.994788 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:59.030434 (+ 35646us) log.cc:842] Serialized 26 byte log entry
0929 14:11:59.254140 (+223706us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:59.254196 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:59.254362 (+ 166us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:59.341799 (+ 87437us) raft_consensus.cc:1671] finished
0929 14:11:59.341828 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.342012 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":21067,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":103,"thread_start_us":208724,"threads_started":1,"wal-append.queue_time_us":303213}
I20260929 14:11:59.433885 5239 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:11:59.433933 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 46107.3us
I20260929 14:11:59.433099 5275 rpcz_store.cc:276] 0929 14:11:57.362167 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:58.809069 (+1446902us) service_pool.cc:227] Handling call
0929 14:11:58.818201 (+ 9132us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:58.818460 (+ 259us) raft_consensus.cc:1520] Early marking committed up to index 6
0929 14:11:59.175061 (+356601us) log.cc:842] Serialized 14 byte log entry
0929 14:11:59.384143 (+209082us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:59.384239 (+ 96us) raft_consensus.cc:1631] Marking committed up to 6
0929 14:11:59.384303 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:59.384383 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.384482 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":390474,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":58555,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":564956,"threads_started":3,"wal-append.queue_time_us":177209}
I20260929 14:11:59.435117 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.438607 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.439484 7097 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:59.418943 6587 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:59.563058 5270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1393) took 2038 ms. Trace:
I20260929 14:11:59.436311 6675 raft_consensus.cc:2726] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:11:59.540386 5273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1376) took 2292 ms. Trace:
I20260929 14:11:59.564317 5270 rpcz_store.cc:276] 0929 14:11:57.524384 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.107692 (+1583308us) service_pool.cc:227] Handling call
0929 14:11:59.108090 (+ 398us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:59.124470 (+ 16380us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:59.124517 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:59.127048 (+ 2531us) log.cc:842] Serialized 26 byte log entry
0929 14:11:59.370388 (+243340us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:59.370426 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:59.370480 (+ 54us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:59.562846 (+192366us) raft_consensus.cc:1671] finished
0929 14:11:59.562871 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.562998 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15902,"raft.queue_time_us":16540,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":11392,"thread_start_us":193670,"threads_started":1,"wal-append.queue_time_us":297595}
I20260929 14:11:59.417778 6493 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:59.568140 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:59.518777 5240 service_pool.cc:137] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50770 dropped due to backpressure. The service queue is full; it has 50 items.
I20260929 14:11:59.568820 5167 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:11:59.569897 5197 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 1.
I20260929 14:11:59.506824 5276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1380) took 2200 ms. Trace:
I20260929 14:11:59.572508 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:59.573318 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 143472us
I20260929 14:11:59.574697 5350 raft_consensus.cc:1261] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260929 14:11:59.576202 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 181913us
I20260929 14:11:59.567464 5273 rpcz_store.cc:276] 0929 14:11:57.247820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.248019 (+ 199us) service_pool.cc:227] Handling call
0929 14:11:57.248539 (+ 520us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:57.290858 (+ 42319us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:11:57.290904 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:57.300717 (+ 9813us) log.cc:842] Serialized 26 byte log entry
0929 14:11:58.045498 (+744781us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:11:58.045549 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:58.045635 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:59.220871 (+1175236us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda46efbee 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551 00007fbda46f8509 00007fbda46f8499
0929 14:11:59.514315 (+293444us) raft_consensus.cc:1671] finished
0929 14:11:59.514340 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.540294 (+ 25954us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41762,"raft.queue_time_us":45137,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":138,"spinlock_wait_cycles":2720128,"thread_start_us":744405,"threads_started":1,"wal-append.queue_time_us":1017235}
W20260929 14:11:59.436673 7111 log.cc:924] Time spent T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.145s user 0.001s sys 0.001s
I20260929 14:11:59.572362 5276 rpcz_store.cc:276] 0929 14:11:57.306460 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.306612 (+ 152us) service_pool.cc:227] Handling call
0929 14:11:57.306967 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:57.307193 (+ 226us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:11:57.307728 (+ 535us) log.cc:842] Serialized 14 byte log entry
0929 14:11:57.318018 (+ 10290us) log.cc:842] Serialized 14 byte log entry
0929 14:11:57.847587 (+529569us) spinlock_profiling.cc:243] Waited 1.8 ms on lock 0x7b180016a200. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b45638 00007fbda1b48b11 00007fbda1b48ac9 00007fbda1b48a59 00007fbda1b48a21 00007fbda1b47d1d 00007fbdaada9204 00007fbdaada9039 00007fbda1b27dc6
0929 14:11:59.061501 (+1213914us) spinlock_profiling.cc:243] Waited 4.24 ms on lock 0x7b180016a200. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b45638 00007fbda1b48b11 00007fbda1b48ac9 00007fbda1b48a59 00007fbda1b48a21 00007fbda1b47d1d 00007fbdaada9204 00007fbdaada9039 00007fbda1b27dc6
0929 14:11:59.405936 (+344435us) spinlock_profiling.cc:243] Waited 2.5 ms on lock 0x7b180016a200. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b27345 00007fbdaa95ea39 00007fbda1b448bb 00007fbda1b42cbf 00007fbda1b42155 00007fbda4a653d0 00007fbda4a77576 00007fbda4a880d7 00007fbda4a88058
0929 14:11:59.506258 (+100322us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:59.506310 (+ 52us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:11:59.506364 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:59.506650 (+ 286us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.506766 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8652,"prepare.queue_time_us":1502,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":80,"raft.queue_time_us":88945,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":167,"spinlock_wait_cycles":19638784,"thread_start_us":100201,"threads_started":1}
I20260929 14:11:59.578518 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.582837 5275 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:59.505615 7020 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:59.583649 7020 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:59.585384 5350 raft_consensus.cc:1261] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260929 14:11:59.513540 5899 raft_consensus.cc:1226] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5. Ops: []
I20260929 14:11:59.581557 6984 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:59.587194 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:59.588567 7058 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.513350 6900 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:59.588953 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1392) took 2120 ms. Trace:
I20260929 14:11:59.589353 6900 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.590380 5902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:59.574244 7027 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:59.591391 7027 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:59.592204 5900 raft_consensus.cc:1261] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:11:59.592636 5198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1395) took 1052 ms. Trace:
I20260929 14:11:59.578186 5316 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:59.566126 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:59.577065 5242 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:50768 (request call id 920) took 2109 ms. Trace:
I20260929 14:11:59.592998 5198 rpcz_store.cc:276] 0929 14:11:58.540233 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:58.718486 (+178253us) service_pool.cc:227] Handling call
0929 14:11:59.591972 (+873486us) spinlock_profiling.cc:243] Waited 6.08 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:11:59.592154 (+ 182us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:59.592333 (+ 179us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:59.592358 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:59.592399 (+ 41us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:11:59.592432 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:59.592490 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.592580 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":13973888}
I20260929 14:11:59.596024 5199 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:11:59.589429 5271 rpcz_store.cc:276] 0929 14:11:57.468928 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.032879 (+1563951us) service_pool.cc:227] Handling call
0929 14:11:59.033239 (+ 360us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:59.033451 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:59.033473 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:59.036628 (+ 3155us) log.cc:842] Serialized 26 byte log entry
0929 14:11:59.349447 (+312819us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:59.349489 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:59.349558 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:59.588742 (+239184us) raft_consensus.cc:1671] finished
0929 14:11:59.588780 (+ 38us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.588901 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":294312,"threads_started":1,"wal-append.queue_time_us":376667}
I20260929 14:11:59.596964 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.579421 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:59.598999 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:59.595422 5167 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:59.599397 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.599673 5273 raft_consensus.cc:3037] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:59.598244 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.598922 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:59.600028 5271 raft_consensus.cc:2454] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:11:59.597803 5198 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:59.601480 5871 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:59.601687 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.601126 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.602301 5271 raft_consensus.cc:2454] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:59.602632 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.602646 5350 raft_consensus.cc:2454] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:59.601717 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.598609 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:59.436570 6583 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:59.604158 5239 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:59.600425 5276 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:59.606835 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.502962 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:11:59.576462 7065 consensus_meta.cc:287] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.520s user 0.009s sys 0.005s
I20260929 14:11:59.608888 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:59.609668 5201 raft_consensus.cc:2379] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 5: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:59.611517 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.611891 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.612277 5873 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:11:59.613025 5198 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:11:59.598059 5267 rpcz_store.cc:276] 0929 14:11:56.377666 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:58.556328 (+2178662us) service_pool.cc:227] Handling call
0929 14:11:58.698979 (+142651us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:58.699222 (+ 243us) raft_consensus.cc:1520] Early marking committed up to index 2
0929 14:11:59.062356 (+363134us) log.cc:842] Serialized 14 byte log entry
0929 14:11:59.393775 (+331419us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:59.393826 (+ 51us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:59.393877 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:59.393933 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.394050 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":470345,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":5466,"raft.queue_time_us":56424,"raft.run_cpu_time_us":53,"raft.run_wall_time_us":52,"thread_start_us":659537,"threads_started":3,"wal-append.queue_time_us":342846}
I20260929 14:11:59.614279 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:59.614976 5198 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:59.606513 6915 raft_consensus.cc:2726] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:11:59.617228 5350 raft_consensus.cc:1261] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:11:59.604790 5242 rpcz_store.cc:276] 0929 14:11:57.467980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.577019 (+2109039us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260929 14:11:59.617758 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.574759 6845 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:59.581521 6914 raft_consensus.cc:2781] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:11:59.578276 7040 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:11:59.589015 7079 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.497934 7045 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.593495 7092 raft_consensus.cc:2781] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:11:59.630167 5273 raft_consensus.cc:2454] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:11:59.630374 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.631094 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.611121 6550 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:11:59.632678 5342 raft_consensus.cc:2454] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:11:59.594660 6920 consensus_queue.cc:1035] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260929 14:11:59.637550 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.639568 7008 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:59.616739 6984 raft_consensus.cc:2781] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:59.441365 7049 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:59.642292 5871 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:11:59.642437 7027 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.643585 6751 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.658659 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.616765 7036 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:11:59.574549 6922 consensus_queue.cc:237] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.636592 5197 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:11:59.625039 6914 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:59.646622 7092 raft_consensus.cc:686] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:59.650779 5198 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:11:59.646332 5201 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:11:59.436860 7002 consensus_queue.cc:260] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:59.606573 6731 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260929 14:11:59.669660 5347 raft_consensus.cc:2379] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 6: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:59.656229 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:59.655238 7020 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:59.621673 7029 raft_consensus.cc:2781] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260929 14:11:59.678603 6642 consensus_meta.cc:287] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.514s user 0.010s sys 0.000s
I20260929 14:11:59.678689 7045 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.678936 7029 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:11:59.679400 7029 raft_consensus.cc:3037] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:59.622633 6659 consensus_queue.cc:1035] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260929 14:11:59.618829 7065 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.659277 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1394) took 1119 ms. Trace:
I20260929 14:11:59.645116 6493 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.582005 5240 service_pool.cc:140] UpdateConsensus request on kudu.consensus.ConsensusService from 127.0.0.1:50770 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1739)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1723)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50776 (request call id 1420)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1727)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1732)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1735)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1744)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1740)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1731)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1725)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1737)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1741)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1733)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1743)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1736)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48196 (request call id 8)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 170)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 171)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 172)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 173)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1395)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1396)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 174)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1397)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1398)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1399)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1400)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 176)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1406)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1408)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1409)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1414)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1415)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1416)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1418)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1419)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 179)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1421)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1422)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50770 (request call id 1738)
I20260929 14:11:59.662150 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.666949 5345 raft_consensus.cc:2454] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:59.685073 5197 raft_consensus.cc:2454] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:11:59.685441 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:59.687428 7092 consensus_queue.cc:237] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:59.688807 6751 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:59.690933 6812 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:11:59.668444 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:59.629405 6928 consensus_queue.cc:1035] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.029s
I20260929 14:11:59.606675 7058 consensus_queue.cc:1046] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260929 14:11:59.669684 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.661128 7132 raft_consensus.cc:2781] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:11:59.666078 6914 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:59.656946 6984 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:59.670080 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:59.677902 6987 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:59.571101 6587 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:11:59.702015 7072 log.cc:924] Time spent T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 1.318s user 0.002s sys 0.003s
I20260929 14:11:59.703147 6587 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:59.675633 5269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1381) took 2356 ms. Trace:
I20260929 14:11:59.704504 7093 consensus_queue.cc:1035] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:11:59.703717 5269 rpcz_store.cc:276] 0929 14:11:57.319787 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.374598 (+ 54811us) service_pool.cc:227] Handling call
0929 14:11:57.374937 (+ 339us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:57.415466 (+ 40529us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:57.415509 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:58.956988 (+1541479us) spinlock_profiling.cc:243] Waited 44.9 ms on lock 0x7b2c0004c1b8. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda3450e11 00007fbda3460442 00007fbda4aa8c11 00007fbda4a6d58a 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772
0929 14:11:58.957490 (+ 502us) log.cc:842] Serialized 26 byte log entry
0929 14:11:59.182827 (+225337us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:59.182879 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:59.182976 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:59.665851 (+482875us) raft_consensus.cc:1671] finished
0929 14:11:59.665870 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.675560 (+ 9690us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":49059,"raft.queue_time_us":40695,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":17540,"spinlock_wait_cycles":103235840,"thread_start_us":224685,"threads_started":1,"wal-append.queue_time_us":333697}
I20260929 14:11:59.705597 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.705641 5907 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45248 (request call id 282) took 2562 ms. Trace:
I20260929 14:11:59.706224 5269 raft_consensus.cc:2454] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:59.707074 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.707870 5269 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:59.706225 5907 rpcz_store.cc:276] 0929 14:11:57.142954 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.143829 (+ 875us) service_pool.cc:227] Handling call
0929 14:11:57.175921 (+ 32092us) raft_consensus.cc:1457] Updating replica for 2 ops
0929 14:11:57.195866 (+ 19945us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:57.195899 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0929 14:11:57.201384 (+ 5485us) log.cc:842] Serialized 52 byte log entry
0929 14:11:58.043220 (+841836us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:11:58.098642 (+ 55422us) log.cc:842] Serialized 14 byte log entry
0929 14:11:58.098859 (+ 217us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:58.098924 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:59.586052 (+1487128us) spinlock_profiling.cc:243] Waited 4.74 ms on lock 0x7b1800060700. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b45638 00007fbda1b48b11 00007fbda1b48ac9 00007fbda1b48a59 00007fbda1b48a21 00007fbda1b47d1d 00007fbdaada9204 00007fbdaada9039 00007fbda1b27dc6
0929 14:11:59.701458 (+115406us) spinlock_profiling.cc:243] Waited 3.57 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda46efbee 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551 00007fbda46f8509 00007fbda46f8499
0929 14:11:59.705320 (+ 3862us) raft_consensus.cc:1671] finished
0929 14:11:59.705354 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.705547 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":56502,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":3856,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":10772,"spinlock_wait_cycles":19109632,"thread_start_us":807825,"threads_started":1,"wal-append.queue_time_us":1174000}
I20260929 14:11:59.679075 6642 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.709775 5270 raft_consensus.cc:2454] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:11:59.710791 6642 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.711589 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.712661 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.714564 5270 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:59.715497 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.715560 6493 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:59.691823 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:59.716156 5270 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:11:59.717095 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.717620 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.718966 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.719039 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:59.719769 5270 raft_consensus.cc:2454] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:59.719897 5201 raft_consensus.cc:3037] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:59.665481 6917 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.720769 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.721654 5270 raft_consensus.cc:2396] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260929 14:11:59.722038 7049 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:59.722672 5270 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:48196
I20260929 14:11:59.722961 6917 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:11:59.723258 5270 raft_consensus.cc:595] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:11:59.708669 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.723668 7065 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.724072 5269 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:59.721711 5348 raft_consensus.cc:2379] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:59.699739 6731 raft_consensus.cc:3032] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:11:59.724831 6731 raft_consensus.cc:729] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:59.724120 6984 raft_consensus.cc:3037] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:59.725628 6731 consensus_queue.cc:260] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:59.735626 7017 raft_consensus.cc:3032] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:11:59.735962 6731 raft_consensus.cc:3037] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:59.737977 6912 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.738021 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:59.675704 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.739929 5192 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:11:59.740788 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.741554 5192 raft_consensus.cc:2379] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 4: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:59.682493 5196 rpcz_store.cc:276] 0929 14:11:58.539957 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:58.709990 (+170033us) service_pool.cc:227] Handling call
0929 14:11:59.504223 (+794233us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:11:59.504407 (+ 184us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:59.504617 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 2
0929 14:11:59.504907 (+ 290us) log.cc:842] Serialized 14 byte log entry
0929 14:11:59.658863 (+153956us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:59.658930 (+ 67us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:59.658992 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:59.659085 (+ 93us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.659204 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8719,"prepare.queue_time_us":8304,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":7727,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":5226,"spinlock_wait_cycles":33553152,"thread_start_us":145942,"threads_started":1,"wal-append.queue_time_us":167382}
I20260929 14:11:59.742825 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.726667 5270 raft_consensus.cc:603] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:11:59.692196 5198 raft_consensus.cc:2379] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 6: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:59.736045 7017 raft_consensus.cc:729] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:11:59.744132 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:59.727985 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.691414 6812 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:59.744700 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:59.828640 5197 raft_consensus.cc:2454] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 1.
I20260929 14:11:59.757987 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:11:59.807739 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 57024.4us
I20260929 14:11:59.761898 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:59.830554 5196 raft_consensus.cc:2379] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 5: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:59.831300 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.760229 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:59.798977 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:59.762981 5314 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:11:59.832950 5351 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:59.829519 5271 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:59.834457 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.834997 5271 raft_consensus.cc:2454] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:11:59.833408 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:11:59.835074 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:59.833549 5199 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 7
I20260929 14:11:59.836920 5344 raft_consensus.cc:2379] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 5: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
W20260929 14:11:59.836891 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 73991.9us
I20260929 14:11:59.837430 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:59.838303 5196 raft_consensus.cc:2379] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 4: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:59.836967 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:59.839299 5198 raft_consensus.cc:2379] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 4: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:11:59.843918 5351 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:11:59.844769 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.845072 5164 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:11:59.846063 7092 raft_consensus.cc:2781] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:11:59.845840 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:11:59.846235 5084 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.GetConsensusState -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 86349.7us
I20260929 14:11:59.846645 7092 raft_consensus.cc:686] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:11:59.846782 5271 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:11:59.848230 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:11:59.853150 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.853981 5351 raft_consensus.cc:2379] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:59.854763 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:11:59.854717 5197 raft_consensus.cc:2379] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:11:59.855069 5314 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
W20260929 14:11:59.845664 6732 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.461s user 0.008s sys 0.009s
I20260929 14:11:59.857309 7017 consensus_queue.cc:260] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:11:59.859061 6732 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.475s user 0.008s sys 0.009s
I20260929 14:11:59.859442 7017 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:11:59.859742 5200 raft_consensus.cc:1226] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260929 14:11:59.847935 7020 raft_consensus.cc:2781] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:11:59.860572 7020 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:11:59.860932 7020 raft_consensus.cc:3037] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:11:59.860945 5200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1390) took 1390 ms. Trace:
I20260929 14:11:59.861588 5200 rpcz_store.cc:276] 0929 14:11:58.470483 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:58.471123 (+ 640us) service_pool.cc:227] Handling call
0929 14:11:58.471538 (+ 415us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:59.860178 (+1388640us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:59.860713 (+ 535us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54003dbb54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:59.860887 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3191949184}
I20260929 14:11:59.791088 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:11:59.862640 6984 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:11:59.863914 5269 raft_consensus.cc:1226] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 3. Current term is 4. Ops: []
I20260929 14:11:59.864848 6984 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:11:59.865108 5269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 170) took 2340 ms. Trace:
I20260929 14:11:59.865654 5269 rpcz_store.cc:276] 0929 14:11:57.525103 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.726557 (+2201454us) service_pool.cc:227] Handling call
0929 14:11:59.737669 (+ 11112us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:59.864322 (+126653us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:59.864857 (+ 535us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54001bf3d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:11:59.865041 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":289912192}
I20260929 14:11:59.867043 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:11:59.867743 5269 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:11:59.713234 7132 raft_consensus.cc:686] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [term 5 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:11:59.870790 5314 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:59.871806 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:11:59.872824 5344 raft_consensus.cc:2379] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
W20260929 14:11:59.858698 7084 log.cc:924] Time spent T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.923s user 0.000s sys 0.001s
I20260929 14:11:59.862694 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:59.764885 6900 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.754205 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:11:59.780229 6912 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:11:59.999632 7036 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:11:59.875043 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1397) took 1334 ms. Trace:
I20260929 14:12:00.002333 7036 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:00.002534 5196 rpcz_store.cc:276] 0929 14:11:58.540769 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.874102 (+1333333us) service_pool.cc:227] Handling call
0929 14:11:59.874453 (+ 351us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:59.874657 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:59.874690 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:59.874748 (+ 58us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:59.874797 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:59.874860 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.874977 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.005837 7036 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:11:59.922891 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:00.012483 5196 raft_consensus.cc:1261] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:12:00.014537 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1401) took 1472 ms. Trace:
I20260929 14:12:00.014881 5196 rpcz_store.cc:276] 0929 14:11:58.541840 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.004510 (+1462670us) service_pool.cc:227] Handling call
0929 14:12:00.012108 (+ 7598us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.014307 (+ 2199us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.014487 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.016182 5196 raft_consensus.cc:1261] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:11:59.925495 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:00.017940 5270 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
W20260929 14:11:59.984400 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 113664us
I20260929 14:11:59.926092 5198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1398) took 1385 ms. Trace:
I20260929 14:12:00.022918 5198 rpcz_store.cc:276] 0929 14:11:58.541037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.875343 (+1334306us) service_pool.cc:227] Handling call
0929 14:11:59.875681 (+ 338us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:59.925735 (+ 50054us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:59.925764 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:59.925808 (+ 44us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:59.925844 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:59.925904 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.926037 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62157,"raft.queue_time_us":50521,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":9581}
I20260929 14:12:00.030937 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1405) took 1488 ms. Trace:
I20260929 14:12:00.031602 5196 rpcz_store.cc:276] 0929 14:11:58.542847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.015572 (+1472725us) service_pool.cc:227] Handling call
0929 14:12:00.015854 (+ 282us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.016460 (+ 606us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.030869 (+ 14409us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.033248 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1408) took 1489 ms. Trace:
I20260929 14:12:00.033686 5196 rpcz_store.cc:276] 0929 14:11:58.543616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.032351 (+1488735us) service_pool.cc:227] Handling call
0929 14:12:00.032678 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.032862 (+ 184us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.032887 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.032928 (+ 41us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:12:00.032961 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.033012 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.033178 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.034186 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:00.039994 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:00.039994 5269 raft_consensus.cc:2454] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:00.042680 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:00.041693 5271 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:00.046348 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:00.047039 5199 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:00.049577 5196 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:12:00.051069 5199 raft_consensus.cc:1226] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 1. Current term is 3. Ops: []
I20260929 14:12:00.052757 5270 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:11:59.945596 6926 raft_consensus.cc:2726] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:00.059548 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1408) took 1009 ms. Trace:
I20260929 14:12:00.110065 5199 rpcz_store.cc:276] 0929 14:11:59.050027 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.050506 (+1000479us) service_pool.cc:227] Handling call
0929 14:12:00.050853 (+ 347us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.059198 (+ 8345us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.059456 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.110989 5270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 172) took 2539 ms. Trace:
I20260929 14:11:59.836704 7007 consensus_queue.cc:1046] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.206s
I20260929 14:11:59.926991 7041 raft_consensus.cc:2781] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260929 14:12:00.105731 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 55338.1us
I20260929 14:12:00.106367 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1401) took 1164 ms. Trace:
I20260929 14:12:00.108148 5201 raft_consensus.cc:2454] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:11:59.947831 5268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1385) took 2584 ms. Trace:
I20260929 14:12:00.109251 5200 raft_consensus.cc:2379] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 6: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:00.052821 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 171) took 2496 ms. Trace:
I20260929 14:12:00.053918 5871 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:11:59.902174 5905 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45252 (request call id 428) took 2160 ms (client timeout 2100 ms). Trace:
I20260929 14:12:00.114737 5196 rpcz_store.cc:276] 0929 14:11:58.941804 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.047870 (+1106066us) service_pool.cc:227] Handling call
0929 14:12:00.048402 (+ 532us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.050627 (+ 2225us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.106295 (+ 55668us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.038924 6642 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:00.118185 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:00.120121 5269 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4b277670ba5344b493c046b2a0864597"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:48196
I20260929 14:12:00.119351 5270 rpcz_store.cc:276] 0929 14:11:57.571258 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.109808 (+2538550us) service_pool.cc:227] Handling call
0929 14:12:00.110312 (+ 504us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.110565 (+ 253us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.110595 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.110653 (+ 58us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:00.110702 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.110775 (+ 73us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.110924 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.279258 5199 raft_consensus.cc:1226] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: [4.2-4.2]
I20260929 14:12:00.279438 5269 raft_consensus.cc:595] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:00.279945 5269 raft_consensus.cc:603] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:12:00.181723 5271 rpcz_store.cc:276] 0929 14:11:57.556004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.051883 (+2495879us) service_pool.cc:227] Handling call
0929 14:12:00.052215 (+ 332us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.052418 (+ 203us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.052455 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.052509 (+ 54us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:00.052570 (+ 61us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.052637 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.052755 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.279956 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1409) took 1195 ms. Trace:
I20260929 14:12:00.281844 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1395) took 2548 ms. Trace:
I20260929 14:11:59.926615 6941 consensus_queue.cc:1046] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.186s
I20260929 14:12:00.250087 5196 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260929 14:12:00.250483 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 196641us
I20260929 14:12:00.290388 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 171) took 1092 ms. Trace:
I20260929 14:12:00.290840 5196 rpcz_store.cc:276] 0929 14:11:59.198396 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.249192 (+1050796us) service_pool.cc:227] Handling call
0929 14:12:00.249616 (+ 424us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.290083 (+ 40467us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.290322 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.292402 5196 raft_consensus.cc:1261] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260929 14:12:00.116645 5201 raft_consensus.cc:1261] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260929 14:12:00.175606 5873 leader_election.cc:334] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.135s (SENT)
I20260929 14:12:00.294054 5873 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:11:59.949934 6629 consensus_queue.cc:1046] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.162s
I20260929 14:12:00.284577 5276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1396) took 2530 ms. Trace:
I20260929 14:12:00.296047 5276 rpcz_store.cc:276] 0929 14:11:57.753913 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.283326 (+2529413us) service_pool.cc:227] Handling call
0929 14:12:00.283710 (+ 384us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.284124 (+ 414us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.284164 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.284216 (+ 52us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:00.284263 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.284333 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.284514 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":98}
I20260929 14:12:00.298408 5276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 174) took 2507 ms. Trace:
I20260929 14:12:00.282491 5271 rpcz_store.cc:276] 0929 14:11:57.733863 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.280921 (+2547058us) service_pool.cc:227] Handling call
0929 14:12:00.281241 (+ 320us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.281449 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.281485 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.281549 (+ 64us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:00.281598 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.281665 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.281780 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.288734 5200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 169) took 1090 ms. Trace:
I20260929 14:11:59.922343 6914 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:00.281926 5199 rpcz_store.cc:276] 0929 14:11:59.084552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.111328 (+1026776us) service_pool.cc:227] Handling call
0929 14:12:00.278977 (+167649us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:00.279689 (+ 712us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.279894 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.300447 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1397) took 2500 ms. Trace:
I20260929 14:12:00.311143 5271 rpcz_store.cc:276] 0929 14:11:57.800183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.299321 (+2499138us) service_pool.cc:227] Handling call
0929 14:12:00.299608 (+ 287us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.300024 (+ 416us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.300064 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.300138 (+ 74us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:00.300186 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.300258 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.300384 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.293147 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1415) took 1031 ms. Trace:
I20260929 14:12:00.313526 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1398) took 2446 ms. Trace:
I20260929 14:12:00.309237 6642 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:00.278558 7121 raft_consensus.cc:2726] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:00.250985 7087 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:00.315343 6900 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:00.315478 7087 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:00.315850 6900 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:00.316298 6642 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:00.317505 7087 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:00.317963 6900 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:00.318393 6731 consensus_meta.cc:287] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.582s user 0.011s sys 0.000s
W20260929 14:12:00.247721 5905 rpcz_store.cc:269] 0929 14:11:57.703489 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.706314 (+ 2825us) service_pool.cc:227] Handling call
0929 14:11:59.790930 (+2084616us) spinlock_profiling.cc:243] Waited 1.33 ms on lock 0x7b5800027698. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0dbce 000000000034aea9 00007fbda9939a30 00007fbda99ff7b4 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:11:59.863711 (+ 72781us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3050880}
I20260929 14:12:00.320281 5269 raft_consensus.cc:1226] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:00.320420 6914 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:00.320911 5269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1394) took 2587 ms. Trace:
I20260929 14:12:00.321388 5269 rpcz_store.cc:276] 0929 14:11:57.733156 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.280653 (+2547497us) service_pool.cc:227] Handling call
0929 14:12:00.320119 (+ 39466us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.320679 (+ 560us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.320847 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.313939 5271 rpcz_store.cc:276] 0929 14:11:57.867446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.312599 (+2445153us) service_pool.cc:227] Handling call
0929 14:12:00.312900 (+ 301us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.313074 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.313158 (+ 84us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.313200 (+ 42us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:00.313260 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.313317 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.313473 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.324028 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1400) took 2325 ms. Trace:
I20260929 14:12:00.324375 5271 rpcz_store.cc:276] 0929 14:11:57.998562 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.322878 (+2324316us) service_pool.cc:227] Handling call
0929 14:12:00.323541 (+ 663us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.323698 (+ 157us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.323771 (+ 73us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.323806 (+ 35us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:00.323846 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.323891 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.323979 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.325994 5270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 173) took 2659 ms. Trace:
I20260929 14:12:00.326138 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 176) took 1908 ms. Trace:
I20260929 14:12:00.326370 5270 rpcz_store.cc:276] 0929 14:11:57.667017 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.279546 (+2612529us) service_pool.cc:227] Handling call
0929 14:12:00.325508 (+ 45962us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.325676 (+ 168us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.325699 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.325735 (+ 36us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:00.325787 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.325837 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.325945 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.326447 5271 rpcz_store.cc:276] 0929 14:11:58.417473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.325401 (+1907928us) service_pool.cc:227] Handling call
0929 14:12:00.325622 (+ 221us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.325848 (+ 226us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.325870 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.325906 (+ 36us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:00.325950 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.325997 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.326081 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.328346 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:00.329376 5346 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1495) took 1147 ms. Trace:
I20260929 14:12:00.329766 5346 rpcz_store.cc:276] 0929 14:11:59.181782 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.181897 (+ 115us) service_pool.cc:227] Handling call
0929 14:12:00.328168 (+1146271us) spinlock_profiling.cc:243] Waited 1.33 ms on lock 0x7b5800183998. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:12:00.329309 (+ 1141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3056000}
I20260929 14:12:00.302662 5276 rpcz_store.cc:276] 0929 14:11:57.791351 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.297471 (+2506120us) service_pool.cc:227] Handling call
0929 14:12:00.297814 (+ 343us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.298027 (+ 213us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.298064 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.298118 (+ 54us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:00.298161 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.298226 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.298346 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.251191 6907 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:00.313334 5196 rpcz_store.cc:276] 0929 14:11:59.261907 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.291724 (+1029817us) service_pool.cc:227] Handling call
0929 14:12:00.292076 (+ 352us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.292761 (+ 685us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.293080 (+ 319us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.033908 6912 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:00.121178 7041 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:00.363270 5271 raft_consensus.cc:1261] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:11:59.970439 7092 consensus_queue.cc:237] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:11:59.982435 7020 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:00.251996 7058 raft_consensus.cc:3032] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:00.379602 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1408) took 1445 ms. Trace:
I20260929 14:12:00.295534 5201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1410) took 1210 ms. Trace:
I20260929 14:12:00.385758 5201 rpcz_store.cc:276] 0929 14:11:59.084894 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.115828 (+1030934us) service_pool.cc:227] Handling call
0929 14:12:00.116229 (+ 401us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.295221 (+178992us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.295468 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.384301 5271 rpcz_store.cc:276] 0929 14:11:58.934375 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.362311 (+1427936us) service_pool.cc:227] Handling call
0929 14:12:00.362808 (+ 497us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.379287 (+ 16479us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.379535 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.388940 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1414) took 1341 ms. Trace:
I20260929 14:11:59.922138 7132 consensus_queue.cc:237] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:00.395903 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1417) took 1117 ms. Trace:
I20260929 14:12:00.400169 5199 rpcz_store.cc:276] 0929 14:11:59.279037 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.310816 (+1031779us) service_pool.cc:227] Handling call
0929 14:12:00.395448 (+ 84632us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.395621 (+ 173us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.395644 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.395684 (+ 40us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:00.395715 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.395765 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.395854 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.396262 5195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1382) took 3262 ms. Trace:
I20260929 14:12:00.415092 5195 rpcz_store.cc:276] 0929 14:11:57.133859 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.802743 (+668884us) service_pool.cc:227] Handling call
0929 14:11:57.803042 (+ 299us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:11:57.825526 (+ 22484us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:57.825555 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:11:57.825594 (+ 39us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:11:57.825631 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.396057 (+2570426us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b5400351454. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:00.396098 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.396202 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51329792}
I20260929 14:12:00.420996 5201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1420) took 1039 ms. Trace:
I20260929 14:12:00.421407 5201 rpcz_store.cc:276] 0929 14:11:59.381608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.387344 (+1005736us) service_pool.cc:227] Handling call
0929 14:12:00.420578 (+ 33234us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.420699 (+ 121us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.420714 (+ 15us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.420738 (+ 24us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:00.420758 (+ 20us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.420826 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.420923 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.382426 7041 raft_consensus.cc:3037] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:00.392601 7058 raft_consensus.cc:729] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:00.375594 6907 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:00.251442 6917 consensus_queue.cc:1035] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.130s
I20260929 14:12:00.376470 6629 consensus_queue.cc:1225] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Peer ac17d3b553b74c9da39e0ab307addacd log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260929 14:12:00.332244 6915 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:00.389477 5271 rpcz_store.cc:276] 0929 14:11:59.047218 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.388076 (+1340858us) service_pool.cc:227] Handling call
0929 14:12:00.388430 (+ 354us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.388615 (+ 185us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.388637 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.388672 (+ 35us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:12:00.388703 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.388754 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.388879 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.398778 7043 consensus_queue.cc:1035] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.036s
I20260929 14:12:00.434186 6812 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:00.522300 5195 raft_consensus.cc:1226] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260929 14:12:00.504590 7019 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.030s sys 0.039s
I20260929 14:12:00.527882 7020 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:00.530414 5272 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1697) took 4649 ms (client timeout 1939 ms). Trace:
W20260929 14:12:00.530879 5272 rpcz_store.cc:269] 0929 14:11:55.880477 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:55.921505 (+ 41028us) service_pool.cc:227] Handling call
0929 14:12:00.530200 (+4608695us) spinlock_profiling.cc:243] Waited 2.82 s on lock 0x7b54003377d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:00.530307 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6487199488}
I20260929 14:12:00.520581 6941 raft_consensus.cc:2726] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260929 14:12:00.517102 5192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1413) took 1250 ms. Trace:
I20260929 14:12:00.534184 5192 rpcz_store.cc:276] 0929 14:11:59.261334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.288460 (+1027126us) service_pool.cc:227] Handling call
0929 14:12:00.391715 (+103255us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.405969 (+ 14254us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.430056 (+ 24087us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.440252 (+ 10196us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:00.459194 (+ 18942us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.495172 (+ 35978us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.511352 (+ 16180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:00.504882 6915 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:00.538359 6915 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:00.526589 7019 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.529s user 0.031s sys 0.039s
W20260929 14:12:00.541373 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 365805us
I20260929 14:12:00.572459 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:00.544881 5901 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45252 (request call id 429) took 2734 ms. Trace:
I20260929 14:12:00.559290 6914 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:00.587265 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 296379us
I20260929 14:12:00.605463 5198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1407) took 2062 ms. Trace:
I20260929 14:12:00.638739 5198 rpcz_store.cc:276] 0929 14:11:58.543378 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.024409 (+1481031us) service_pool.cc:227] Handling call
0929 14:12:00.024847 (+ 438us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:00.037872 (+ 13025us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.037900 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:00.038325 (+ 425us) log.cc:842] Serialized 26 byte log entry
0929 14:12:00.416697 (+378372us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:00.416748 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.416839 (+ 91us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:00.605235 (+188396us) raft_consensus.cc:1671] finished
0929 14:12:00.605254 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.605401 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12108,"raft.queue_time_us":13502,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":7777,"thread_start_us":378049,"threads_started":1,"wal-append.queue_time_us":565033}
I20260929 14:12:00.636039 5314 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:00.120064 5268 rpcz_store.cc:276] 0929 14:11:57.361805 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.767221 (+405416us) service_pool.cc:227] Handling call
0929 14:11:57.785875 (+ 18654us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:57.847412 (+ 61537us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:57.847453 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:57.847999 (+ 546us) log.cc:842] Serialized 26 byte log entry
0929 14:11:58.213170 (+365171us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:11:58.213269 (+ 99us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:58.255446 (+ 42177us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:11:59.858157 (+1602711us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda46efbee 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551 00007fbda46f8509 00007fbda46f8499
0929 14:11:59.926359 (+ 68202us) raft_consensus.cc:1671] finished
0929 14:11:59.938787 (+ 12428us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:11:59.945725 (+ 6938us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1057,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":119,"spinlock_wait_cycles":55807232,"thread_start_us":343057,"threads_started":1,"wal-append.queue_time_us":953294}
W20260929 14:12:00.594215 7142 log.cc:924] Time spent T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.194s user 0.000s sys 0.001s
W20260929 14:12:00.603204 7085 log.cc:924] Time spent T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 1.672s user 0.005s sys 0.016s
W20260929 14:12:00.610163 7138 log.cc:924] Time spent T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.219s user 0.000s sys 0.001s
W20260929 14:12:00.608502 7017 consensus_meta.cc:287] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.748s user 0.010s sys 0.001s
I20260929 14:12:00.676676 5314 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:00.682358 5314 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:00.590059 6912 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:00.676889 6907 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:00.687613 5314 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:00.678766 6915 raft_consensus.cc:2781] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:00.583891 6914 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:00.686261 7020 raft_consensus.cc:2781] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election won for term 6
W20260929 14:12:00.697690 7029 consensus_meta.cc:287] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 1.006s user 0.012s sys 0.000s
I20260929 14:12:00.688117 6868 raft_consensus.cc:2726] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:12:00.694936 7017 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:12:00.697875 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:00.700220 6915 raft_consensus.cc:686] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:00.708384 7020 raft_consensus.cc:686] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:00.753925 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1422) took 1134 ms. Trace:
I20260929 14:12:00.754510 5199 rpcz_store.cc:276] 0929 14:11:59.619370 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.443861 (+824491us) service_pool.cc:227] Handling call
0929 14:12:00.506489 (+ 62628us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.506730 (+ 241us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:12:00.507014 (+ 284us) log.cc:842] Serialized 14 byte log entry
0929 14:12:00.753592 (+246578us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.753640 (+ 48us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:00.753684 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.753751 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.753865 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":104,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"thread_start_us":227197,"threads_started":1}
I20260929 14:12:00.753758 7018 consensus_queue.cc:1035] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:12:00.722023 7017 raft_consensus.cc:686] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:00.708009 7029 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:00.762914 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 638776us
I20260929 14:12:00.765332 5906 raft_consensus.cc:1226] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:00.577912 5901 rpcz_store.cc:276] 0929 14:11:57.810321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:57.810562 (+ 241us) service_pool.cc:227] Handling call
0929 14:11:57.810936 (+ 374us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:57.905910 (+ 94974us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:11:57.905941 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:57.906483 (+ 542us) log.cc:842] Serialized 26 byte log entry
0929 14:11:58.291619 (+385136us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:11:58.299550 (+ 7931us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.458372 (+2158822us) spinlock_profiling.cc:243] Waited 37.7 ms on lock 0x7b5400488354. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:00.461080 (+ 2708us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:00.499457 (+ 38377us) raft_consensus.cc:1671] finished
0929 14:12:00.499481 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.543507 (+ 44026us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22856,"raft.queue_time_us":53880,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":15007,"spinlock_wait_cycles":86645120,"thread_start_us":374990,"threads_started":1,"wal-append.queue_time_us":523259,"wal-append.run_cpu_time_us":1908,"wal-append.run_wall_time_us":1001920}
I20260929 14:12:00.768283 6915 consensus_queue.cc:237] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:00.773113 5346 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 1827) took 1060 ms. Trace:
I20260929 14:12:00.773607 5346 rpcz_store.cc:276] 0929 14:11:59.712555 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.375706 (+663151us) service_pool.cc:227] Handling call
0929 14:12:00.376127 (+ 421us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:00.382036 (+ 5909us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.382066 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:00.382582 (+ 516us) log.cc:842] Serialized 26 byte log entry
0929 14:12:00.674460 (+291878us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:00.674502 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.674565 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:00.772869 (+ 98304us) raft_consensus.cc:1671] finished
0929 14:12:00.772889 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.773044 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5493,"raft.queue_time_us":33088,"spinlock_wait_cycles":4453632,"thread_start_us":283892,"threads_started":1,"wal-append.queue_time_us":375992}
I20260929 14:12:00.775766 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:00.776811 5346 raft_consensus.cc:2354] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:00.774605 5316 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:00.781773 6868 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:00.782233 5239 leader_election.cc:398] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:00.782323 6868 raft_consensus.cc:686] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:00.782598 5239 leader_election.cc:401] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260929 14:12:00.783486 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:00.783337 6868 consensus_queue.cc:237] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:00.785597 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:00.787045 7018 consensus_queue.cc:1046] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:12:00.787184 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:00.787942 6917 raft_consensus.cc:3032] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:00.785413 6912 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
W20260929 14:12:00.790328 5345 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1498) took 1434 ms (client timeout 1508 ms). Trace:
W20260929 14:12:00.804208 5345 rpcz_store.cc:269] 0929 14:11:59.356317 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.370302 (+ 13985us) service_pool.cc:227] Handling call
0929 14:12:00.790139 (+1419837us) spinlock_profiling.cc:243] Waited 248 ms on lock 0x7b540027db54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:00.790260 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":569840384}
I20260929 14:12:00.806037 5239 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:00.788290 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:00.788882 7018 consensus_queue.cc:1035] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260929 14:12:00.813848 6946 raft_consensus.cc:2781] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:12:00.814303 6946 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:00.814718 6946 raft_consensus.cc:3037] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:00.793054 7083 consensus_queue.cc:1035] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260929 14:12:00.788296 6917 raft_consensus.cc:729] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:00.817130 6917 consensus_queue.cc:260] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:00.823666 7150 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:00.823890 7041 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:00.824275 7150 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:00.825361 7017 consensus_queue.cc:237] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:00.826968 7150 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:00.828008 7154 log.cc:924] Time spent T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260929 14:12:00.829123 5269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1399) took 2854 ms. Trace:
I20260929 14:12:00.829633 5269 rpcz_store.cc:276] 0929 14:11:57.974350 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.322268 (+2347918us) service_pool.cc:227] Handling call
0929 14:12:00.322686 (+ 418us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:00.323135 (+ 449us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.323175 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:00.323690 (+ 515us) log.cc:842] Serialized 26 byte log entry
0929 14:12:00.579630 (+255940us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:12:00.579684 (+ 54us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.579774 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:00.828856 (+249082us) raft_consensus.cc:1671] finished
0929 14:12:00.828884 (+ 28us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.829055 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":112,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":125,"thread_start_us":255550,"threads_started":1,"wal-append.queue_time_us":400777}
W20260929 14:12:00.854276 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 213693us
I20260929 14:12:00.864189 5346 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:00.865104 5346 raft_consensus.cc:595] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:00.873442 7036 raft_consensus.cc:3032] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:12:00.873868 7036 raft_consensus.cc:729] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:00.874686 7036 consensus_queue.cc:260] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:00.876389 5348 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47252 (request call id 1814) took 1642 ms (client timeout 2155 ms). Trace:
W20260929 14:12:00.876852 5348 rpcz_store.cc:269] 0929 14:11:59.234111 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.255678 (+ 21567us) service_pool.cc:227] Handling call
0929 14:12:00.876181 (+1620503us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b5400251cd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:00.876328 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":740916608}
I20260929 14:12:00.878378 5348 raft_consensus.cc:1226] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:00.878945 5348 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 147) took 1139 ms. Trace:
I20260929 14:12:00.879220 6914 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:00.880448 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:00.883450 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:00.927909 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:00.774688 7020 consensus_queue.cc:237] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:00.929875 6946 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:00.942695 5269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1419) took 1732 ms. Trace:
I20260929 14:12:00.943655 5269 rpcz_store.cc:276] 0929 14:11:59.210233 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.884786 (+1674553us) service_pool.cc:227] Handling call
0929 14:12:00.942105 (+ 57319us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.942317 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.942351 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.942403 (+ 52us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:00.942449 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.942513 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.942635 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:11:59.618904 5242 service_pool.cc:140] GetConsensusState request on kudu.consensus.ConsensusService from 127.0.0.1:50768 dropped due to backpressure. The service queue is full; it has 50 items. Contents of service queue:
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1727)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1732)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1735)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1744)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1740)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1731)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1725)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1737)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1741)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1733)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1743)
Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1736)
Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:48196 (request call id 8)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 170)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 171)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 172)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 173)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1394)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1395)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1396)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 174)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1397)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1398)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1399)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1400)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 176)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1406)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1408)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1409)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1414)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1415)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1416)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1418)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1419)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 179)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1421)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1422)
Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50770 (request call id 1738)
I20260929 14:12:00.299211 5200 rpcz_store.cc:276] 0929 14:11:59.197823 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.182745 (+984922us) service_pool.cc:227] Handling call
0929 14:12:00.183141 (+ 396us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.288294 (+105153us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.288335 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.288388 (+ 53us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:12:00.288451 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.288531 (+ 80us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:00.288669 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":104751,"raft.queue_time_us":106099}
I20260929 14:12:01.104408 5346 raft_consensus.cc:1226] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:01.250739 7010 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:01.110350 6717 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:01.085386 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:01.253285 7010 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:01.254263 5193 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37270 (request call id 170) took 2056 ms (client timeout 2135 ms). Trace:
I20260929 14:12:01.174911 5903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45248 (request call id 286) took 2636 ms. Trace:
I20260929 14:12:01.183266 5270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1406) took 2249 ms. Trace:
W20260929 14:12:01.254747 5193 rpcz_store.cc:269] 0929 14:11:59.198110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.674013 (+475903us) service_pool.cc:227] Handling call
0929 14:12:01.074951 (+1400938us) spinlock_profiling.cc:243] Waited 1.51 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:12:01.254203 (+179252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3478400}
I20260929 14:12:01.255426 7010 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:01.140188 5201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1421) took 1527 ms. Trace:
I20260929 14:12:00.879298 5348 rpcz_store.cc:276] 0929 14:11:59.739391 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.877827 (+1138436us) service_pool.cc:227] Handling call
0929 14:12:00.878187 (+ 360us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.878726 (+ 539us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.878892 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:01.161137 5268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1418) took 2008 ms. Trace:
I20260929 14:12:01.191717 7058 consensus_queue.cc:260] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:01.105660 6917 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:00.925468 7036 raft_consensus.cc:3037] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:01.180485 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:01.256464 5346 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 148) took 1516 ms. Trace:
I20260929 14:12:01.185729 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:01.255081 5903 rpcz_store.cc:276] 0929 14:11:58.538516 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:58.538928 (+ 412us) service_pool.cc:227] Handling call
0929 14:11:58.539340 (+ 412us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:58.539847 (+ 507us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:11:58.539887 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:11:58.540477 (+ 590us) log.cc:842] Serialized 26 byte log entry
0929 14:11:59.065701 (+525224us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:11:59.065740 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:11:59.065798 (+ 58us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:01.174056 (+2108258us) spinlock_profiling.cc:243] Waited 4.65 ms on lock 0x7b54007a33a0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda49fb771 00007fbda4a06220 00007fbda4a061c8 00007fbda4a06148 00007fbda4a06103 00007fbda4a0538c 00007fbdaaa76206 00007fbdaaa6e808 00007fbda4a4238b 00007fbda4a44e80
0929 14:12:01.174569 (+ 513us) raft_consensus.cc:1671] finished
0929 14:12:01.174594 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.174836 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2484,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":105,"spinlock_wait_cycles":10702592,"thread_start_us":517129,"threads_started":1,"wal-append.queue_time_us":655625}
W20260929 14:12:01.146354 5343 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1491) took 1991 ms (client timeout 1783 ms). Trace:
I20260929 14:12:01.104897 7152 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:01.126936 5347 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1493) took 1956 ms (client timeout 1793 ms). Trace:
I20260929 14:12:01.265017 7152 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:01.265249 5347 rpcz_store.cc:269] 0929 14:11:59.170706 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.170948 (+ 242us) service_pool.cc:227] Handling call
0929 14:12:01.126711 (+1955763us) spinlock_profiling.cc:243] Waited 494 ms on lock 0x7b54002526d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:01.126866 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1136850688}
I20260929 14:12:01.248543 5198 raft_consensus.cc:1261] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:12:01.273814 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:01.111111 5272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1416) took 2059 ms. Trace:
I20260929 14:12:01.194662 5276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1409) took 2254 ms. Trace:
I20260929 14:12:01.210974 5098 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52750 (request call id 221) took 8592 ms. Trace:
I20260929 14:12:01.164045 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:01.276121 5273 raft_consensus.cc:2379] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 5: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
W20260929 14:12:01.276736 5273 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1727) took 1859 ms (client timeout 2070 ms). Trace:
W20260929 14:12:01.277313 5273 rpcz_store.cc:269] 0929 14:11:59.417888 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.163610 (+1745722us) service_pool.cc:227] Handling call
0929 14:12:01.276684 (+113074us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":89600}
I20260929 14:12:01.278234 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:01.138134 5906 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45232 (request call id 352) took 1338 ms. Trace:
I20260929 14:12:01.278555 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:01.279264 5198 raft_consensus.cc:2454] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:12:01.278544 5242 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:01.260226 5270 rpcz_store.cc:276] 0929 14:11:58.933773 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.345873 (+1412100us) service_pool.cc:227] Handling call
0929 14:12:00.396651 (+ 50778us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.410119 (+ 13468us) raft_consensus.cc:1520] Early marking committed up to index 5
0929 14:12:00.725522 (+315403us) log.cc:842] Serialized 14 byte log entry
0929 14:12:01.104797 (+379275us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:01.104872 (+ 75us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:01.104936 (+ 64us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.183007 (+ 78071us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b5400393354. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:01.183064 (+ 57us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.183199 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":418916,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":6067,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"spinlock_wait_cycles":30479616,"thread_start_us":559035,"threads_started":2,"wal-append.queue_time_us":380216}
I20260929 14:12:01.107070 7002 raft_consensus.cc:3037] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:01.112291 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:01.305426 5267 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 4
I20260929 14:12:01.022293 5269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 179) took 1791 ms. Trace:
I20260929 14:12:01.306317 5269 rpcz_store.cc:276] 0929 14:11:59.230849 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.944863 (+1714014us) service_pool.cc:227] Handling call
0929 14:12:01.021682 (+ 76819us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:01.021903 (+ 221us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:01.021939 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:01.021994 (+ 55us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:01.022042 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.022106 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.022219 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260929 14:12:01.159554 5342 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1500) took 1584 ms (client timeout 1836 ms). Trace:
W20260929 14:12:01.308171 5342 rpcz_store.cc:269] 0929 14:11:59.575195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.576006 (+ 811us) service_pool.cc:227] Handling call
0929 14:12:01.159350 (+1583344us) spinlock_profiling.cc:243] Waited 29.3 ms on lock 0x7b540012cd54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:01.159467 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":67325696}
I20260929 14:12:01.195295 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:01.267400 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1419) took 1897 ms. Trace:
I20260929 14:12:01.119060 5344 raft_consensus.cc:3037] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:01.309619 5196 rpcz_store.cc:276] 0929 14:11:59.369649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.379142 (+1009493us) service_pool.cc:227] Handling call
0929 14:12:00.379521 (+ 379us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.379756 (+ 235us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:12:00.762576 (+382820us) log.cc:842] Serialized 14 byte log entry
0929 14:12:01.267102 (+504526us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:01.267150 (+ 48us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:01.267200 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.267262 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.267350 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":183386,"prepare.queue_time_us":14985,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":22528,"raft.queue_time_us":189665,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":5624,"thread_start_us":698537,"threads_started":2,"wal-append.queue_time_us":336202}
I20260929 14:12:01.086439 7035 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:01.262591 5346 rpcz_store.cc:276] 0929 14:11:59.739787 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.103795 (+1364008us) service_pool.cc:227] Handling call
0929 14:12:01.104197 (+ 402us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:01.256245 (+152048us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.256413 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:01.320843 5346 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 153) took 1315 ms. Trace:
I20260929 14:12:01.321305 5346 rpcz_store.cc:276] 0929 14:12:00.005597 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.319883 (+1314286us) service_pool.cc:227] Handling call
0929 14:12:01.320251 (+ 368us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:01.320471 (+ 220us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:01.320507 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:01.320573 (+ 66us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:01.320626 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.320693 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.320791 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:01.323333 5346 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47258 (request call id 1506) took 1317 ms. Trace:
I20260929 14:12:01.323731 5346 rpcz_store.cc:276] 0929 14:12:00.005949 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.322419 (+1316470us) service_pool.cc:227] Handling call
0929 14:12:01.322760 (+ 341us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:01.322959 (+ 199us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:01.322995 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:01.323044 (+ 49us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:01.323088 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.323158 (+ 70us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.323275 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:01.258651 7058 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:01.270511 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:01.255957 5201 rpcz_store.cc:276] 0929 14:11:59.612754 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.422123 (+809369us) service_pool.cc:227] Handling call
0929 14:12:00.495815 (+ 73692us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.512639 (+ 16824us) raft_consensus.cc:1520] Early marking committed up to index 2
0929 14:12:00.526083 (+ 13444us) log.cc:842] Serialized 14 byte log entry
0929 14:12:00.767033 (+240950us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:00.767082 (+ 49us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:00.767125 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.139284 (+372159us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b180014b4e0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b12f96 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9
0929 14:12:01.139926 (+ 642us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b54001469d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:01.139980 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.140124 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4489,"prepare.queue_time_us":28616,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":52,"raft.queue_time_us":4584,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":2293,"spinlock_wait_cycles":42644608,"thread_start_us":232569,"threads_started":1,"wal-append.queue_time_us":425538}
I20260929 14:12:01.258248 5348 raft_consensus.cc:1261] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260929 14:12:01.274763 5276 rpcz_store.cc:276] 0929 14:11:58.940727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.346390 (+1405663us) service_pool.cc:227] Handling call
0929 14:12:00.347252 (+ 862us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.347794 (+ 542us) raft_consensus.cc:1520] Early marking committed up to index 2
0929 14:12:00.759844 (+412050us) log.cc:842] Serialized 14 byte log entry
0929 14:12:01.180800 (+420956us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:01.180853 (+ 53us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:01.180899 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.180962 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.194600 (+ 13638us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":445530,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":106,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"thread_start_us":784183,"threads_started":2,"wal-append.queue_time_us":385431}
I20260929 14:12:01.278978 5906 rpcz_store.cc:276] 0929 14:11:59.799658 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.799897 (+ 239us) service_pool.cc:227] Handling call
0929 14:11:59.800224 (+ 327us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.768896 (+968672us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.137833 (+368937us) spinlock_profiling.cc:243] Waited 965 ms on lock 0x7b54006fed54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:01.138058 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2219025664}
I20260929 14:12:01.211570 5097 catalog_manager.cc:5495] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 1 to 3, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 3 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:01.338913 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:01.339766 5097 catalog_manager.cc:5495] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 1 to 2, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 2 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:01.123320 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1415) took 2075 ms. Trace:
I20260929 14:12:01.280939 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:01.342358 5198 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:01.285661 5273 raft_consensus.cc:3037] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:01.287400 6989 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:01.352941 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 62917.5us
I20260929 14:12:01.353827 5345 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:01.354380 5345 raft_consensus.cc:595] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:01.341744 5271 rpcz_store.cc:276] 0929 14:11:59.047776 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.503150 (+1455374us) service_pool.cc:227] Handling call
0929 14:12:00.503559 (+ 409us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:00.503976 (+ 417us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.504015 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:00.509969 (+ 5954us) log.cc:842] Serialized 26 byte log entry
0929 14:12:00.768320 (+258351us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:12:00.768358 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.768420 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:01.123109 (+354689us) raft_consensus.cc:1671] finished
0929 14:12:01.123133 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.123259 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":93,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":14840,"thread_start_us":206979,"threads_started":1,"wal-append.queue_time_us":511379}
I20260929 14:12:01.354696 5345 raft_consensus.cc:603] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:12:01.127951 6946 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:01.048615 7148 log.cc:924] Time spent T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.323s user 0.001s sys 0.000s
I20260929 14:12:01.095633 6912 raft_consensus.cc:2726] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:01.304520 5242 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:01.352131 5097 catalog_manager.cc:5495] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 1 to 3, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 3 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260929 14:12:01.378567 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 74124.4us
I20260929 14:12:01.164027 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:01.312390 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:01.379243 5242 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:01.379599 5274 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:01.267663 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:01.276026 5098 rpcz_store.cc:276] 0929 14:11:52.618446 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:52.618780 (+ 334us) service_pool.cc:227] Handling call
0929 14:12:01.210848 (+8592068us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:11:56.046032 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:11:56.060733 (+ 14701us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:11:56.060775 (+ 42us) write_op.cc:435] Acquired schema lock
0929 14:11:56.060796 (+ 21us) tablet.cc:660] Decoding operations
0929 14:11:56.061828 (+ 1032us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:11:56.061914 (+ 86us) write_op.cc:641] Partition lock acquired for write op
0929 14:11:56.061944 (+ 30us) tablet.cc:683] Acquiring locks for 8 operations
0929 14:11:56.062593 (+ 649us) tablet.cc:699] Row locks acquired
0929 14:11:56.062613 (+ 20us) write_op.cc:260] PREPARE: finished
0929 14:11:56.062743 (+ 130us) write_op.cc:270] Start()
0929 14:11:56.062839 (+ 96us) write_op.cc:276] Timestamp: P: 1790691116062703 usec, L: 0
0929 14:11:56.062859 (+ 20us) op_driver.cc:331] REPLICATION: starting
0929 14:11:56.097480 (+ 34621us) log.cc:842] Serialized 3281 byte log entry
0929 14:11:56.740040 (+642560us) op_driver.cc:444] REPLICATION: finished
0929 14:11:57.870799 (+1130759us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda1b275fe 00007fbdaa95ea39 00007fbda1b448bb 00007fbda1b42cbf 00007fbda1b44e22 00007fbda847d410
0929 14:11:58.935316 (+1064517us) write_op.cc:301] APPLY: starting
0929 14:11:58.938296 (+ 2980us) tablet.cc:1365] starting BulkCheckPresence
0929 14:11:58.982960 (+ 44664us) tablet.cc:1368] finished BulkCheckPresence
0929 14:11:58.988271 (+ 5311us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:11:59.210038 (+221767us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:11:59.291527 (+ 81489us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0929 14:11:59.291591 (+ 64us) write_op.cc:312] APPLY: finished
0929 14:11:59.294462 (+ 2871us) log.cc:842] Serialized 65 byte log entry
0929 14:11:59.344242 (+ 49780us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:59.344416 (+ 174us) write_op.cc:454] Released schema lock
0929 14:11:59.344481 (+ 65us) write_op.cc:341] FINISH: Updating metrics
0929 14:11:59.344889 (+ 408us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:11:59.344930 (+ 41us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1252,"reported_tablets":26,"child_traces":[["op",{"apply.queue_time_us":2060036,"apply.run_cpu_time_us":14268,"apply.run_wall_time_us":522686,"num_ops":8,"prepare.queue_time_us":270177,"prepare.run_cpu_time_us":4316,"prepare.run_wall_time_us":78386,"replication_time_us":669726,"spinlock_wait_cycles":6649600,"thread_start_us":1928967,"threads_started":3,"wal-append.queue_time_us":45516,"wal-append.run_cpu_time_us":1521,"wal-append.run_wall_time_us":1007293}]]}
W20260929 14:12:01.313808 5275 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1735) took 1892 ms (client timeout 2175 ms). Trace:
W20260929 14:12:01.382045 5275 rpcz_store.cc:269] 0929 14:11:59.421630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.194951 (+1773321us) service_pool.cc:227] Handling call
0929 14:12:01.313738 (+118787us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:01.061419 7041 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:01.316830 5344 raft_consensus.cc:2454] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:01.182857 7029 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:01.286890 6916 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:01.328955 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:01.335253 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:01.336483 5342 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260929 14:12:01.287555 7121 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:01.264712 5343 rpcz_store.cc:269] 0929 14:11:59.154565 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.156653 (+ 2088us) service_pool.cc:227] Handling call
0929 14:12:01.146040 (+1989387us) spinlock_profiling.cc:243] Waited 617 ms on lock 0x7b540012d254. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:01.146262 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1418632448}
I20260929 14:12:01.286994 6751 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:01.299862 7132 consensus_queue.cc:1046] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.016s
I20260929 14:12:01.344856 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:01.274649 5272 rpcz_store.cc:276] 0929 14:11:59.051378 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.531856 (+1480478us) service_pool.cc:227] Handling call
0929 14:12:00.532289 (+ 433us) raft_consensus.cc:1457] Updating replica for 2 ops
0929 14:12:00.532677 (+ 388us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.532707 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0929 14:12:00.541312 (+ 8605us) log.cc:842] Serialized 52 byte log entry
0929 14:12:00.790930 (+249618us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:12:00.790983 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:00.791067 (+ 84us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:01.110878 (+319811us) raft_consensus.cc:1671] finished
0929 14:12:01.110897 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.111052 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":4724,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":88,"thread_start_us":221694,"threads_started":1,"wal-append.queue_time_us":564666}
I20260929 14:12:01.355355 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:01.355942 5345 raft_consensus.cc:1226] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:01.360141 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:01.343799 7152 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:01.367462 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 429545us
I20260929 14:12:01.313253 7128 raft_consensus.cc:2726] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:12:01.372977 5273 raft_consensus.cc:2454] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:12:01.431758 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:01.382665 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 78140.2us
I20260929 14:12:01.383889 5196 raft_consensus.cc:1261] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:12:01.384370 5349 raft_consensus.cc:1226] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:01.433893 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:01.434752 5271 raft_consensus.cc:2379] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:01.435627 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:01.384684 5873 leader_election.cc:334] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.631s (SENT)
I20260929 14:12:01.385728 5242 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:01.378917 5097 catalog_manager.cc:5495] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 2 to 6, leader changed from 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 6 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:01.353356 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:01.417546 5347 raft_consensus.cc:1226] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 3. Current term is 4. Ops: []
W20260929 14:12:01.414867 5344 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1505) took 1488 ms (client timeout 1678 ms). Trace:
I20260929 14:12:01.440526 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:01.438639 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 52983.6us
I20260929 14:12:01.441179 5269 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
W20260929 14:12:01.393790 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 68165.7us
I20260929 14:12:01.416345 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:01.441737 5242 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:12:01.415465 5342 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 152) took 1622 ms. Trace:
I20260929 14:12:01.442309 5342 rpcz_store.cc:276] 0929 14:11:59.793606 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.309127 (+1515521us) service_pool.cc:227] Handling call
0929 14:12:01.336095 (+ 26968us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:01.415197 (+ 79102us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.415403 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:01.424703 5274 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:01.442827 5242 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:01.443449 5274 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50776 (request call id 1420) took 2157 ms (client timeout 2129 ms). Trace:
I20260929 14:12:01.338160 7035 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:01.445341 5873 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:01.387821 6946 raft_consensus.cc:2781] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:01.438644 5267 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
W20260929 14:12:01.443950 5274 rpcz_store.cc:269] 0929 14:11:59.286002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.163667 (+1877665us) service_pool.cc:227] Handling call
0929 14:12:01.443367 (+279700us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:01.445580 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:12:01.440352 5344 rpcz_store.cc:269] 0929 14:11:59.926959 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.039750 (+112791us) service_pool.cc:227] Handling call
0929 14:12:01.414684 (+1374934us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400258fd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:01.414811 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2475650944}
I20260929 14:12:01.387022 5348 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 149) took 1643 ms. Trace:
I20260929 14:12:01.436955 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:01.387511 6914 raft_consensus.cc:2726] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:01.387364 7083 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:01.374327 6989 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:01.449855 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 71971us
I20260929 14:12:01.686903 5345 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:01.449558 6946 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:01.688644 7035 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:01.687950 7083 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:01.446569 6912 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260929 14:12:01.544061 7185 log.cc:924] Time spent T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260929 14:12:01.433808 6751 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:01.691390 5351 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 1826) took 2000 ms. Trace:
I20260929 14:12:01.557662 5197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1399) took 3016 ms. Trace:
I20260929 14:12:01.691933 6751 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:01.452127 7147 log.cc:924] Time spent T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260929 14:12:01.691816 5351 rpcz_store.cc:276] 0929 14:11:59.691183 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.941891 (+250708us) service_pool.cc:227] Handling call
0929 14:11:59.944666 (+ 2775us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:00.006743 (+ 62077us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:00.006796 (+ 53us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:01.151458 (+1144662us) spinlock_profiling.cc:243] Waited 4.76 ms on lock 0x7b2c000339d8. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda3450e11 00007fbda3460442 00007fbda4aa8c11 00007fbda4a6d58a 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772
0929 14:12:01.151896 (+ 438us) log.cc:842] Serialized 26 byte log entry
0929 14:12:01.299358 (+147462us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:01.299408 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.299474 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:01.691062 (+391588us) raft_consensus.cc:1671] finished
0929 14:12:01.691082 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.691329 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":59861,"raft.queue_time_us":60792,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"spinlock_wait_cycles":10951552,"thread_start_us":145841,"threads_started":1,"wal-append.queue_time_us":188141}
W20260929 14:12:01.545275 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 112677us
I20260929 14:12:01.690881 7083 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:01.694415 5271 raft_consensus.cc:1226] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6. Ops: [3.2-5.3]
W20260929 14:12:01.637472 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 187325us
I20260929 14:12:01.687271 6914 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:01.257550 5268 rpcz_store.cc:276] 0929 14:11:59.153262 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.761857 (+1608595us) service_pool.cc:227] Handling call
0929 14:12:00.770209 (+ 8352us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.792198 (+ 21989us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:12:00.879828 (+ 87630us) log.cc:842] Serialized 14 byte log entry
0929 14:12:01.160646 (+280818us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:01.160718 (+ 72us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:01.160784 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.160875 (+ 91us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.161032 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56678,"prepare.queue_time_us":1702,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":14365,"raft.queue_time_us":201,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":83,"thread_start_us":280379,"threads_started":1,"wal-append.queue_time_us":287058}
I20260929 14:12:01.694955 5273 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:01.691955 5197 rpcz_store.cc:276] 0929 14:11:58.541320 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.858832 (+1317512us) service_pool.cc:227] Handling call
0929 14:11:59.901791 (+ 42959us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:11:59.994604 (+ 92813us) raft_consensus.cc:1520] Early marking committed up to index 2
0929 14:12:00.039354 (+ 44750us) log.cc:842] Serialized 14 byte log entry
0929 14:12:00.538242 (+498888us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b180027e260. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b27345 00007fbdaa95ea39 00007fbda1b448bb 00007fbda1b42cbf 00007fbda1b44e22 00007fbda46ee5bb 00007fbda46f4063 00007fbda46f457b 00007fbda4a7788a
0929 14:12:00.945458 (+407216us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:01.051958 (+106500us) log.cc:842] Serialized 26 byte log entry
0929 14:12:01.052134 (+ 176us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:01.052446 (+ 312us) log.cc:842] Serialized 14 byte log entry
0929 14:12:01.446742 (+394296us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.446905 (+ 163us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:01.557261 (+110356us) raft_consensus.cc:1671] finished
0929 14:12:01.557306 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.557569 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":286792,"prepare.queue_time_us":48433,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":100,"raft.queue_time_us":412461,"raft.run_cpu_time_us":458,"raft.run_wall_time_us":77140,"spinlock_wait_cycles":50988672,"thread_start_us":526228,"threads_started":2,"wal-append.queue_time_us":1134939}
I20260929 14:12:01.705475 5270 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:01.711187 5348 rpcz_store.cc:276] 0929 14:11:59.740141 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.257623 (+1517482us) service_pool.cc:227] Handling call
0929 14:12:01.257915 (+ 292us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:01.337371 (+ 79456us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.383456 (+ 46085us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:01.513047 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:01.712874 5272 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:12:01.524706 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:01.715440 5239 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:01.525290 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:01.719131 5342 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:12:01.721050 5269 raft_consensus.cc:3037] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:12:01.449877 5267 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1732) took 2030 ms (client timeout 2117 ms). Trace:
I20260929 14:12:01.687654 5345 raft_consensus.cc:595] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:01.697292 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50770 (request call id 1738) took 2272 ms. Trace:
W20260929 14:12:01.722076 5267 rpcz_store.cc:269] 0929 14:11:59.419411 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.112004 (+1692593us) service_pool.cc:227] Handling call
0929 14:12:01.449818 (+337814us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:01.722536 5271 rpcz_store.cc:276] 0929 14:11:59.422501 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.693285 (+2270784us) service_pool.cc:227] Handling call
0929 14:12:01.694166 (+ 881us) raft_consensus.cc:1457] Updating replica for 2 ops
0929 14:12:01.694921 (+ 755us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.695108 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:01.700896 7128 raft_consensus.cc:2741] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260929 14:12:01.449332 6587 raft_consensus.cc:2781] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:12:01.730963 5870 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:01.678222 7186 log.cc:924] Time spent T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
W20260929 14:12:01.681716 7167 log.cc:924] Time spent T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.354s user 0.000s sys 0.001s
I20260929 14:12:01.689688 6912 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:01.733942 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 291149us
I20260929 14:12:01.747326 5242 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:01.749384 5242 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:01.687690 5316 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:01.686100 7195 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:01.656602 7132 raft_consensus.cc:3032] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:12:01.619925 7191 consensus_queue.cc:1046] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.218s
I20260929 14:12:01.687989 6946 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:01.695575 7152 raft_consensus.cc:2726] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:01.737838 6587 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:01.756549 7198 raft_consensus.cc:2781] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:12:01.757323 7204 consensus_queue.cc:1035] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:01.667191 7188 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.093s
I20260929 14:12:01.422530 6916 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:01.430152 7121 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:01.751258 6912 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:01.750654 7128 raft_consensus.cc:2781] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:01.429569 7149 consensus_queue.cc:1035] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.334s
I20260929 14:12:01.717585 6914 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:01.761379 6403 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:01.439487 5097 catalog_manager.cc:5495] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 3 to 4, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 4 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
W20260929 14:12:01.848980 5167 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.003s sys 0.009s
I20260929 14:12:01.696139 5192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1430) took 1633 ms. Trace:
I20260929 14:12:01.851666 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260929 14:12:01.856874 5200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1439) took 1029 ms. Trace:
I20260929 14:12:01.752621 5242 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:01.872216 5342 raft_consensus.cc:1226] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 2. Current term is 3. Ops: []
W20260929 14:12:01.765511 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 77930.4us
I20260929 14:12:01.620263 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:01.929291 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:01.930455 5276 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
W20260929 14:12:01.932262 7002 consensus_meta.cc:287] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.620s user 0.010s sys 0.001s
I20260929 14:12:01.874794 5348 raft_consensus.cc:1261] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:12:01.773528 6917 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:01.937738 6917 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:01.938320 6917 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:01.796523 6989 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:01.814647 7121 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:01.760275 7132 raft_consensus.cc:729] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
W20260929 14:12:01.942806 5905 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.525s user 0.000s sys 0.020s
I20260929 14:12:01.942966 6989 raft_consensus.cc:2726] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:01.943333 5905 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.553s user 0.000s sys 0.020s
I20260929 14:12:01.782531 7128 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:01.945866 7128 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
W20260929 14:12:01.813180 7182 log.cc:924] Time spent T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.154s user 0.001s sys 0.000s
I20260929 14:12:01.951498 7152 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:01.952322 7189 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:01.953424 7189 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:01.954736 6914 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:01.956605 7189 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:01.765337 7195 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:01.958397 5097 catalog_manager.cc:5495] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 1 to 3, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 3 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:01.773855 7198 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:01.962237 6991 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:01.963048 6991 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:01.876808 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 445409us
W20260929 14:12:01.922953 5167 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.595s user 0.004s sys 0.009s
I20260929 14:12:01.773976 6587 raft_consensus.cc:3037] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:01.929713 5343 raft_consensus.cc:2379] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 3: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:01.738482 5314 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:01.786892 7191 raft_consensus.cc:3032] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:12:01.798372 7208 raft_consensus.cc:2726] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:01.974563 5274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1422) took 2581 ms. Trace:
W20260929 14:12:01.975817 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 791713us
I20260929 14:12:01.975080 5274 rpcz_store.cc:276] 0929 14:11:59.390845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.688658 (+2297813us) service_pool.cc:227] Handling call
0929 14:12:01.689289 (+ 631us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:01.956100 (+266811us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:01.956151 (+ 51us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:01.956229 (+ 78us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:01.956267 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.971881 (+ 15614us) spinlock_profiling.cc:243] Waited 266 ms on lock 0x7b54003368d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:01.971933 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.972120 (+ 187us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":612365056}
I20260929 14:12:01.978873 7202 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:01.978765 6403 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:01.979805 7202 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:01.981565 7205 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:01.913548 5200 rpcz_store.cc:276] 0929 14:12:00.826808 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.253935 (+427127us) service_pool.cc:227] Handling call
0929 14:12:01.254190 (+ 255us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:01.304627 (+ 50437us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:01.304658 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:01.328699 (+ 24041us) log.cc:842] Serialized 26 byte log entry
0929 14:12:01.567158 (+238459us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:12:01.567215 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.567316 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:01.856206 (+288890us) raft_consensus.cc:1671] finished
0929 14:12:01.856256 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.856610 (+ 354us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":44387,"raft.queue_time_us":73119,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":20636,"thread_start_us":180196,"threads_started":1,"wal-append.queue_time_us":426813}
I20260929 14:12:01.961086 7198 raft_consensus.cc:3037] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:01.944317 7132 consensus_queue.cc:260] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:01.982532 7205 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:01.986673 7205 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:01.989820 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 61196.6us
I20260929 14:12:01.971931 7191 raft_consensus.cc:729] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
W20260929 14:12:01.991357 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 252794us
I20260929 14:12:01.984478 7132 raft_consensus.cc:3037] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:12:01.982054 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 597222us
I20260929 14:12:01.932855 5276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1435) took 1820 ms. Trace:
I20260929 14:12:01.999593 5276 rpcz_store.cc:276] 0929 14:12:00.112119 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.931436 (+1819317us) service_pool.cc:227] Handling call
0929 14:12:01.932004 (+ 568us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:01.932407 (+ 403us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:01.932430 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:01.932496 (+ 66us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:01.932535 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.932620 (+ 85us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.932742 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:02.010926 5346 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 1833) took 1474 ms. Trace:
I20260929 14:12:02.011408 5349 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 1813) took 2798 ms. Trace:
I20260929 14:12:02.014364 5347 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 150) took 2273 ms. Trace:
I20260929 14:12:02.014916 5347 rpcz_store.cc:276] 0929 14:11:59.740470 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.295635 (+1555165us) service_pool.cc:227] Handling call
0929 14:12:01.414825 (+119190us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:01.446388 (+ 31563us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.013811 (+567423us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b5400258fd0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6ca09 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:02.014119 (+ 308us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":273249664}
I20260929 14:12:02.004405 7205 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:02.021361 7205 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:02.022300 7152 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:02.022892 7152 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:01.998436 6916 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:02.011478 5346 rpcz_store.cc:276] 0929 14:12:00.536671 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.354443 (+817772us) service_pool.cc:227] Handling call
0929 14:12:01.354744 (+ 301us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:01.544719 (+189975us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:01.544764 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:01.628858 (+ 84094us) log.cc:842] Serialized 26 byte log entry
0929 14:12:01.904193 (+275335us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:01.904333 (+ 140us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.010251 (+105918us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b5400259ed4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:02.010310 (+ 59us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:02.010364 (+ 54us) raft_consensus.cc:1671] finished
0929 14:12:02.010379 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:02.010701 (+ 322us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":56491,"raft.queue_time_us":76310,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":3418,"spinlock_wait_cycles":435829504,"thread_start_us":238904,"threads_started":1,"wal-append.queue_time_us":249038}
I20260929 14:12:02.031118 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:12:02.032869 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260929 14:12:02.037371 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 222190us
I20260929 14:12:02.011942 5349 rpcz_store.cc:276] 0929 14:11:59.212527 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.212693 (+ 166us) service_pool.cc:227] Handling call
0929 14:12:01.126843 (+1914150us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:01.433839 (+306996us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.908516 (+474677us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b54002526d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:02.010974 (+102458us) spinlock_profiling.cc:243] Waited 1.91 s on lock 0x7b54002526d0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6ca09 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:02.011178 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4993247360}
I20260929 14:12:02.037662 6916 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:02.043196 7208 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:02.024751 7205 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:02.060712 5351 raft_consensus.cc:1226] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:02.061424 7002 raft_consensus.cc:2781] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:02.061882 7002 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:02.062088 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 71356.9us
I20260929 14:12:02.062336 5239 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:02.064301 7219 raft_consensus.cc:2781] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:12:02.064944 7219 raft_consensus.cc:686] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:02.056833 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:02.066291 7219 consensus_queue.cc:237] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:02.050827 6916 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:01.927321 5192 rpcz_store.cc:276] 0929 14:12:00.041664 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.537076 (+495412us) service_pool.cc:227] Handling call
0929 14:12:00.560240 (+ 23164us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:00.571730 (+ 11490us) raft_consensus.cc:1520] Early marking committed up to index 5
0929 14:12:00.589904 (+ 18174us) log.cc:842] Serialized 14 byte log entry
0929 14:12:01.538965 (+949061us) log.cc:842] Serialized 14 byte log entry
0929 14:12:01.648445 (+109480us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:01.655906 (+ 7461us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:01.659091 (+ 3185us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.663729 (+ 4638us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:01.675039 (+ 11310us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":137072,"prepare.queue_time_us":36117,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":85,"raft.queue_time_us":226299,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":38383,"spinlock_wait_cycles":2944,"thread_start_us":447783,"threads_started":2,"wal-append.queue_time_us":661670}
I20260929 14:12:02.070693 6916 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:02.068989 7061 consensus_queue.cc:1035] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260929 14:12:02.049667 7128 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:02.049048 5276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1437) took 1415 ms. Trace:
I20260929 14:12:02.062347 7002 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:02.089927 5276 rpcz_store.cc:276] 0929 14:12:00.633867 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:02.004282 (+1370415us) service_pool.cc:227] Handling call
0929 14:12:02.048304 (+ 44022us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:02.048726 (+ 422us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:02.048752 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:02.048789 (+ 37us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:12:02.048821 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.048869 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:02.048974 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:02.054111 5270 raft_consensus.cc:2396] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 5 because replica has last-logged OpId of term: 4 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260929 14:12:02.052377 6403 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:02.077149 5267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1431) took 2035 ms. Trace:
I20260929 14:12:02.070709 7040 raft_consensus.cc:979] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:02.072393 5871 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:02.098754 6642 consensus_queue.cc:1035] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.058s
I20260929 14:12:02.099922 5274 raft_consensus.cc:1261] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:02.100741 5274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1436) took 1467 ms. Trace:
I20260929 14:12:02.100857 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:02.101895 5343 raft_consensus.cc:2454] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
W20260929 14:12:02.104051 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 79597.9us
I20260929 14:12:02.101177 5274 rpcz_store.cc:276] 0929 14:12:00.633489 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:02.039454 (+1405965us) service_pool.cc:227] Handling call
0929 14:12:02.099294 (+ 59840us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:02.100430 (+ 1136us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.100669 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:02.108925 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1440) took 1281 ms. Trace:
W20260929 14:12:02.110615 7218 log.cc:924] Time spent T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.107s user 0.000s sys 0.000s
I20260929 14:12:02.117940 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:02.120571 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:02.125984 7208 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:02.128451 7128 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:02.128007 5268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1433) took 2079 ms. Trace:
I20260929 14:12:02.129642 5873 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:02.132965 7227 log.cc:924] Time spent T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.053s user 0.000s sys 0.011s
I20260929 14:12:02.134194 5270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 182) took 1381 ms. Trace:
I20260929 14:12:02.128842 7208 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:02.098749 7058 consensus_meta.cc:287] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.754s user 0.009s sys 0.002s
W20260929 14:12:02.573976 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 444036us
I20260929 14:12:02.093832 5267 rpcz_store.cc:276] 0929 14:12:00.042110 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.723166 (+1681056us) service_pool.cc:227] Handling call
0929 14:12:01.723544 (+ 378us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:01.724173 (+ 629us) raft_consensus.cc:1520] Early marking committed up to index 5
0929 14:12:02.017967 (+293794us) log.cc:842] Serialized 14 byte log entry
0929 14:12:02.075828 (+ 57861us) log.cc:842] Serialized 14 byte log entry
0929 14:12:02.076458 (+ 630us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:02.076519 (+ 61us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:02.076577 (+ 58us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.076675 (+ 98us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:02.076970 (+ 295us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19390,"prepare.queue_time_us":186162,"prepare.run_cpu_time_us":105,"prepare.run_wall_time_us":106,"raft.queue_time_us":130,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":194,"thread_start_us":265694,"threads_started":2,"wal-append.queue_time_us":52942}
I20260929 14:12:02.109390 5193 rpcz_store.cc:276] 0929 14:12:00.827046 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.255964 (+428918us) service_pool.cc:227] Handling call
0929 14:12:01.256266 (+ 302us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:01.327285 (+ 71019us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:01.332466 (+ 5181us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:01.430006 (+ 97540us) log.cc:842] Serialized 26 byte log entry
0929 14:12:01.800545 (+370539us) spinlock_profiling.cc:243] Waited 9.13 ms on lock 0x7b48000432d8. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda99147f1 00007fbda990b8e7 00007fbda990b77a 00007fbda99ff95f 00007fbda99ff627 00007fbda9a13b2c 00007fbda9a13ad8 00007fbda9a13a58 00007fbda9a13a13 00007fbda9a12cbc 00007fbda8994be6
0929 14:12:02.108065 (+307520us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:02.108166 (+ 101us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.108293 (+ 127us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:02.108357 (+ 64us) raft_consensus.cc:1671] finished
0929 14:12:02.108380 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:02.108711 (+ 331us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":41176,"raft.queue_time_us":71074,"raft.run_cpu_time_us":3722,"raft.run_wall_time_us":468435,"spinlock_wait_cycles":20989312,"thread_start_us":354979,"threads_started":1,"wal-append.queue_time_us":449948}
I20260929 14:12:02.152838 5273 raft_consensus.cc:1226] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
W20260929 14:12:02.145077 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 72891.9us
I20260929 14:12:02.141662 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:02.505407 5350 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 1824) took 2864 ms. Trace:
I20260929 14:12:02.401566 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:02.590034 5270 rpcz_store.cc:276] 0929 14:12:00.752587 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:02.133036 (+1380449us) service_pool.cc:227] Handling call
0929 14:12:02.133542 (+ 506us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:02.133865 (+ 323us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:02.133892 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:02.133948 (+ 56us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:02.133978 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.134028 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:02.134133 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:02.392935 5276 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 6.
I20260929 14:12:02.221179 5272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1430) took 2201 ms. Trace:
I20260929 14:12:02.153348 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:12:02.584134 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 454667us
I20260929 14:12:02.399883 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:12:02.351006 6717 consensus_meta.cc:287] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 1.097s user 0.013s sys 0.005s
W20260929 14:12:02.574227 6912 consensus_meta.cc:287] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.525s user 0.007s sys 0.007s
I20260929 14:12:02.614374 5273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1434) took 2495 ms. Trace:
I20260929 14:12:02.227380 6868 raft_consensus.cc:979] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:02.611644 5198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1443) took 1332 ms. Trace:
I20260929 14:12:02.152341 6991 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:02.626271 5274 raft_consensus.cc:1226] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:02.627211 5274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 183) took 1871 ms. Trace:
I20260929 14:12:02.627637 5274 rpcz_store.cc:276] 0929 14:12:00.755869 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:02.625599 (+1869730us) service_pool.cc:227] Handling call
0929 14:12:02.626020 (+ 421us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:02.626891 (+ 871us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.627130 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260929 14:12:02.628832 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 876110us
I20260929 14:12:02.630926 5242 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:02.631596 7002 consensus_meta.cc:287] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.524s user 0.007s sys 0.002s
I20260929 14:12:02.632222 7002 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:02.634511 5201 raft_consensus.cc:1261] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:02.218948 5905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45232 (request call id 354) took 1653 ms. Trace:
I20260929 14:12:02.214210 7149 raft_consensus.cc:1067] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:02.633625 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:02.632627 5242 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:02.221709 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:02.227159 7209 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:02.519888 6914 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:02.400202 7189 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:02.647269 5194 raft_consensus.cc:1261] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:02.199146 7121 raft_consensus.cc:2781] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:12:02.676846 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:02.219410 7202 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:02.234422 7199 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:02.705993 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:02.656163 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:02.593551 5350 rpcz_store.cc:276] 0929 14:11:59.638335 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:59.639033 (+ 698us) service_pool.cc:227] Handling call
0929 14:11:59.639294 (+ 261us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:01.971313 (+2332019us) raft_consensus.cc:1520] Early marking committed up to index 5
0929 14:12:01.971812 (+ 499us) log.cc:842] Serialized 14 byte log entry
0929 14:12:02.106180 (+134368us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:02.106218 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:02.106260 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.153347 (+ 47087us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b48000429d8. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda99147f1 00007fbda990b8e7 00007fbda990b77a 00007fbda99ff95f 00007fbda99ff627 00007fbda9a13b2c 00007fbda9a13ad8 00007fbda9a13a58 00007fbda9a13a13 00007fbda9a12cbc 00007fbda8994be6
0929 14:12:02.502772 (+349425us) spinlock_profiling.cc:243] Waited 2.29 s on lock 0x7b54003a9dd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:02.502840 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:02.503065 (+ 225us) inbound_call.cc:173] Queueing success response
0929 14:12:02.520798 (+ 17733us) spinlock_profiling.cc:243] Waited 4.76 ms on lock 0x7b18002f55a0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b45638 00007fbda1b48b11 00007fbda1b48ac9 00007fbda1b48a59 00007fbda1b48a21 00007fbda1b47d1d 00007fbdaada9204 00007fbdaada9039 00007fbda1b27dc6
Metrics: {"mutex_wait_us":89423,"prepare.queue_time_us":776,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":47,"raft.queue_time_us":52151,"raft.run_cpu_time_us":1425,"raft.run_wall_time_us":45075,"spinlock_wait_cycles":5296278144,"thread_start_us":118120,"threads_started":1,"wal-append.queue_time_us":136644}
I20260929 14:12:02.523154 7152 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:02.229918 7193 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:02.147244 6562 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:02.720949 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:02.708369 7132 consensus_meta.cc:287] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.669s user 0.012s sys 0.000s
I20260929 14:12:02.621145 5273 rpcz_store.cc:276] 0929 14:12:00.107225 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.873830 (+1766605us) service_pool.cc:227] Handling call
0929 14:12:02.095870 (+222040us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:02.582856 (+486986us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.595009 (+ 12153us) spinlock_profiling.cc:243] Waited 220 ms on lock 0x7b540007d850. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6ca09 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:02.602967 (+ 7958us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":505251584}
I20260929 14:12:02.633478 7219 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:02.152561 6917 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:02.622238 5198 rpcz_store.cc:276] 0929 14:12:01.279532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.343028 (+ 63496us) service_pool.cc:227] Handling call
0929 14:12:01.398073 (+ 55045us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:12:01.398408 (+ 335us) raft_consensus.cc:1457] Updating replica for 2 ops
0929 14:12:01.850742 (+452334us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:01.850834 (+ 92us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0929 14:12:01.852279 (+ 1445us) log.cc:842] Serialized 52 byte log entry
0929 14:12:01.915197 (+ 62918us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:12:01.915247 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:01.915442 (+ 195us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:02.364621 (+449179us) raft_consensus.cc:1671] finished
0929 14:12:02.364648 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:02.611443 (+246795us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12085,"raft.queue_time_us":257084,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":5752,"spinlock_wait_cycles":30675840,"thread_start_us":501776,"threads_started":2,"wal-append.queue_time_us":62925}
I20260929 14:12:02.692509 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:02.672190 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:02.147598 7029 consensus_queue.cc:1046] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.570s
I20260929 14:12:02.149106 6587 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:02.723383 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:02.151556 7205 consensus_queue.cc:1046] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.567s
I20260929 14:12:02.652729 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:02.597545 5272 rpcz_store.cc:276] 0929 14:12:00.019899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.713680 (+1693781us) service_pool.cc:227] Handling call
0929 14:12:01.873479 (+159799us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:01.874240 (+ 761us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:12:01.967036 (+ 92796us) log.cc:842] Serialized 14 byte log entry
0929 14:12:02.077530 (+110494us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:02.077577 (+ 47us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:02.077615 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.077706 (+ 91us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:02.220800 (+143094us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54002fd0d0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6ca09 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:02.220948 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":37035,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":420,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":162,"spinlock_wait_cycles":283050752,"thread_start_us":123653,"threads_started":2,"wal-append.queue_time_us":101026}
I20260929 14:12:02.318876 7198 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:02.218506 6946 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:02.744951 5351 raft_consensus.cc:1226] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:02.722990 7058 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:02.642343 7041 consensus_queue.cc:1035] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.087s
I20260929 14:12:02.733749 7209 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:02.742472 6922 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:02.713258 6912 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:02.772203 5342 raft_consensus.cc:1226] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:02.746753 7121 raft_consensus.cc:686] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:02.774657 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:02.777278 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:02.778041 5201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1445) took 1495 ms. Trace:
I20260929 14:12:02.750653 7208 raft_consensus.cc:2781] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:12:02.780220 5201 rpcz_store.cc:276] 0929 14:12:01.282564 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.371675 (+ 89111us) service_pool.cc:227] Handling call
0929 14:12:01.371926 (+ 251us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:02.669496 (+1297570us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.768940 (+ 99444us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b540031d654. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:02.777975 (+ 9035us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2823628672}
I20260929 14:12:02.782107 6907 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:02.787734 5276 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:02.791857 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:02.794047 6822 raft_consensus.cc:1067] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:02.794225 5275 raft_consensus.cc:1226] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:02.789815 5267 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:02.786818 6907 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:02.802703 5164 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:02.808749 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 89224.7us
I20260929 14:12:02.800971 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1432) took 2626 ms. Trace:
I20260929 14:12:02.786965 6922 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:02.814543 5348 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:12:02.787513 7205 consensus_queue.cc:1225] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer d886fb1eb0474d2d9f7410c4013c45e4 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260929 14:12:02.787680 7195 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:02.769143 7189 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:02.777194 7058 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:02.788389 7208 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:02.820741 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 149818us
I20260929 14:12:02.749953 7199 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:02.826924 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:02.129964 5268 rpcz_store.cc:276] 0929 14:12:00.048279 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.742677 (+1694398us) service_pool.cc:227] Handling call
0929 14:12:01.745659 (+ 2982us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:01.745870 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:12:02.025324 (+279454us) log.cc:842] Serialized 14 byte log entry
0929 14:12:02.126277 (+100953us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:02.126412 (+ 135us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:02.126720 (+ 308us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.126922 (+ 202us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:02.127583 (+ 661us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11325,"prepare.queue_time_us":165281,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":78,"raft.queue_time_us":11476,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":144,"thread_start_us":294522,"threads_started":2}
I20260929 14:12:02.830325 5344 raft_consensus.cc:1261] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
W20260929 14:12:02.815441 7221 log.cc:924] Time spent T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.294s user 0.004s sys 0.000s
I20260929 14:12:02.748873 6562 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:02.835990 7041 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:02.810935 5271 rpcz_store.cc:276] 0929 14:12:00.045450 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.723713 (+1678263us) service_pool.cc:227] Handling call
0929 14:12:01.723993 (+ 280us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:01.724169 (+ 176us) raft_consensus.cc:1520] Early marking committed up to index 2
0929 14:12:02.103429 (+379260us) log.cc:842] Serialized 14 byte log entry
0929 14:12:02.459418 (+355989us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:02.495207 (+ 35789us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:02.600339 (+105132us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.671476 (+ 71137us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:02.671644 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":185566,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":81,"raft.queue_time_us":17033,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":13751,"thread_start_us":522265,"threads_started":2,"wal-append.queue_time_us":640826}
I20260929 14:12:02.755731 6914 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:02.840715 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:02.838014 5903 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:02.843690 5199 raft_consensus.cc:1261] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:12:02.844657 5272 raft_consensus.cc:1226] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:02.758896 7193 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:02.845513 5272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50770 (request call id 1755) took 1586 ms. Trace:
I20260929 14:12:02.845809 5272 rpcz_store.cc:276] 0929 14:12:01.258916 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:02.775799 (+1516883us) service_pool.cc:227] Handling call
0929 14:12:02.815721 (+ 39922us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:02.845204 (+ 29483us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.845427 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:02.844043 7191 consensus_queue.cc:260] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:02.830276 7208 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:12:02.840457 7205 raft_consensus.cc:2781] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:02.854341 5906 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:02.858830 5270 raft_consensus.cc:1226] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260929 14:12:02.859761 5270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50770 (request call id 1748) took 1602 ms. Trace:
I20260929 14:12:02.860157 5270 rpcz_store.cc:276] 0929 14:12:01.256960 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:02.772977 (+1516017us) service_pool.cc:227] Handling call
0929 14:12:02.806209 (+ 33232us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:02.859387 (+ 53178us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.859665 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:02.862107 5276 raft_consensus.cc:1226] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260929 14:12:02.862171 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:02.857370 5269 consensus_meta.cc:287] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 1.044s user 0.012s sys 0.007s
I20260929 14:12:02.868120 5276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50770 (request call id 1758) took 1608 ms. Trace:
I20260929 14:12:02.868511 5276 rpcz_store.cc:276] 0929 14:12:01.259608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:02.808502 (+1548894us) service_pool.cc:227] Handling call
0929 14:12:02.840302 (+ 31800us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:02.867843 (+ 27541us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.868061 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260929 14:12:02.872242 5349 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.723s user 0.063s sys 0.159s
W20260929 14:12:02.872644 5349 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.724s user 0.063s sys 0.159s
I20260929 14:12:02.851291 7002 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:02.877436 5270 raft_consensus.cc:1226] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:02.871760 5198 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:12:02.807782 7209 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:02.865374 5314 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:02.878196 5269 raft_consensus.cc:2454] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:12:02.856886 7205 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:02.881588 5275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1421) took 3489 ms. Trace:
I20260929 14:12:02.836632 7041 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:02.887849 5314 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:02.891570 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:02.873318 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:02.873499 6946 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:02.885174 5275 rpcz_store.cc:276] 0929 14:11:59.387416 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.448134 (+2060718us) service_pool.cc:227] Handling call
0929 14:12:01.448509 (+ 375us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:02.804726 (+1356217us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.871302 (+ 66576us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54002fdad4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:02.877078 (+ 5776us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3034940160}
I20260929 14:12:02.900090 5269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1753) took 1637 ms. Trace:
I20260929 14:12:02.900842 5269 rpcz_store.cc:276] 0929 14:12:01.258390 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:01.441740 (+183350us) service_pool.cc:227] Handling call
0929 14:12:02.891653 (+1449913us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:02.887118 7205 raft_consensus.cc:3037] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
W20260929 14:12:02.906484 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 71015.2us
I20260929 14:12:02.871414 7219 consensus_queue.cc:1035] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20260929 14:12:02.886255 7152 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:02.890087 7189 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:02.890813 6917 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:02.914983 5275 raft_consensus.cc:1226] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:02.911010 5272 raft_consensus.cc:1261] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:12:02.916656 5272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1442) took 1644 ms. Trace:
I20260929 14:12:02.917255 5272 rpcz_store.cc:276] 0929 14:12:01.272105 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:02.855932 (+1583827us) service_pool.cc:227] Handling call
0929 14:12:02.884368 (+ 28436us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:02.916315 (+ 31947us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.916585 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:02.910864 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:02.920430 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:02.929934 5268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1445) took 1648 ms. Trace:
I20260929 14:12:02.930510 5268 rpcz_store.cc:276] 0929 14:12:01.281684 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:02.867812 (+1586128us) service_pool.cc:227] Handling call
0929 14:12:02.899154 (+ 31342us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:02.929428 (+ 30274us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:02.929474 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:02.929539 (+ 65us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:02.929584 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.929704 (+ 120us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:02.929859 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:02.932721 5268 raft_consensus.cc:1261] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:12:02.933727 7058 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:02.939224 5272 raft_consensus.cc:1226] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:02.711817 5905 rpcz_store.cc:276] 0929 14:12:00.565580 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:00.568069 (+ 2489us) service_pool.cc:227] Handling call
0929 14:12:01.133562 (+565493us) spinlock_profiling.cc:243] Waited 1.44 ms on lock 0x7b5800027698. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0db7c 00007fbda1b0d73b 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:12:01.133828 (+ 266us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:01.266291 (+132463us) raft_consensus.cc:1520] Early marking committed up to index 6
0929 14:12:01.266607 (+ 316us) log.cc:842] Serialized 14 byte log entry
0929 14:12:01.819223 (+552616us) spinlock_profiling.cc:243] Waited 7.51 ms on lock 0x7b180023c020. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b45638 00007fbda1b48b11 00007fbda1b48ac9 00007fbda1b48a59 00007fbda1b48a21 00007fbda1b47d1d 00007fbdaada9204 00007fbdaada9039 00007fbda1b27dc6
0929 14:12:01.944003 (+124780us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:01.944095 (+ 92us) raft_consensus.cc:1631] Marking committed up to 6
0929 14:12:01.944232 (+ 137us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.218462 (+274230us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b54000cbcd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:02.218525 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:02.218754 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":3243,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":231445,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":6179,"spinlock_wait_cycles":324654464,"thread_start_us":658567,"threads_started":2,"wal-append.queue_time_us":504870,"wal-append.run_cpu_time_us":1510,"wal-append.run_wall_time_us":1030440}
I20260929 14:12:02.942829 7040 consensus_queue.cc:1046] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260929 14:12:02.946524 7040 raft_consensus.cc:3032] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:12:02.947016 7040 raft_consensus.cc:729] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:02.947120 6922 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:02.905077 5192 raft_consensus.cc:1261] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260929 14:12:02.924679 5267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 184) took 1655 ms. Trace:
I20260929 14:12:02.959064 5267 rpcz_store.cc:276] 0929 14:12:01.269294 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:02.811316 (+1542022us) service_pool.cc:227] Handling call
0929 14:12:02.863641 (+ 52325us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:02.914786 (+ 51145us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:02.914826 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:02.924323 (+ 9497us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:02.924379 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.924456 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:02.924603 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:02.943977 5276 raft_consensus.cc:1226] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:02.941169 5272 raft_consensus.cc:1261] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:12:02.902383 7198 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:02.954720 5239 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:02.962736 5270 raft_consensus.cc:1226] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:02.969151 7202 raft_consensus.cc:3032] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:02.948277 7040 consensus_queue.cc:260] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:02.969791 7202 raft_consensus.cc:729] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:02.913414 6912 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:02.901768 6587 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:02.977367 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:02.958681 5192 pending_rounds.cc:76] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd: Aborting all ops after (but not including) 2
I20260929 14:12:02.982955 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:02.982051 5192 pending_rounds.cc:97] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260929 14:12:02.986593 5192 raft_consensus.cc:2866] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:12:02.988050 7202 consensus_queue.cc:260] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:02.989517 5272 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:12:02.990535 7040 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:02.992450 7209 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:02.992595 5272 raft_consensus.cc:1261] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260929 14:12:02.993824 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:03.003140 5271 raft_consensus.cc:1261] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:12:03.007021 5270 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:02.944339 6917 consensus_queue.cc:1035] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260929 14:12:03.010697 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1444) took 1731 ms. Trace:
I20260929 14:12:03.014293 5271 rpcz_store.cc:276] 0929 14:12:01.272783 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:02.864244 (+1591461us) service_pool.cc:227] Handling call
0929 14:12:02.983519 (+119275us) spinlock_profiling.cc:243] Waited 1.92 ms on lock 0x7b58001e5498. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:12:02.983782 (+ 263us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:03.004211 (+ 20429us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:03.004455 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4424064}
I20260929 14:12:02.989543 7202 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:03.027122 5272 raft_consensus.cc:1261] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260929 14:12:02.971567 6946 consensus_queue.cc:1046] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260929 14:12:03.006681 6912 raft_consensus.cc:2781] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:12:02.985296 7152 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.047s
I20260929 14:12:03.044773 5267 raft_consensus.cc:1261] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:03.005270 7208 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:02.907565 7121 consensus_queue.cc:237] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:03.035975 6912 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:03.087743 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 455201us
W20260929 14:12:03.091001 6642 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.935s user 0.007s sys 0.007s
I20260929 14:12:03.054049 7199 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:03.096406 5275 raft_consensus.cc:1226] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:12:03.097821 5270 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:03.053769 7193 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:03.084987 6912 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:12:03.094314 6642 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.938s user 0.007s sys 0.007s
I20260929 14:12:03.133383 5873 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.102113 7041 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:03.141305 5871 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.005s sys 0.012s
W20260929 14:12:03.146605 5871 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.521s user 0.005s sys 0.012s
W20260929 14:12:03.147385 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 521794us
I20260929 14:12:03.150053 5314 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:03.156425 5242 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.123986 6922 consensus_queue.cc:1046] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.023s
W20260929 14:12:03.164413 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 325048us
I20260929 14:12:03.153983 5871 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.152256 6914 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:03.160167 6991 raft_consensus.cc:2781] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260929 14:12:03.163542 7230 log.cc:924] Time spent T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.232s user 0.001s sys 0.000s
I20260929 14:12:03.145838 6917 consensus_queue.cc:1046] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.033s
W20260929 14:12:03.186488 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 441194us
I20260929 14:12:03.173468 7209 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:03.177014 7041 raft_consensus.cc:2781] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:03.193457 7239 raft_consensus.cc:2726] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260929 14:12:03.193302 5084 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet f84af62485a4405f887bb8675221641e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica c5ec116e39144881bdf1f2cd3e923cf5 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 5 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:03.193914 7234 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260929 14:12:03.197057 7232 log.cc:924] Time spent T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
I20260929 14:12:03.197393 5344 raft_consensus.cc:1226] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5. Ops: []
W20260929 14:12:03.197669 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 105603us
I20260929 14:12:03.199601 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:03.200162 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:03.160498 6717 consensus_queue.cc:1035] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
W20260929 14:12:03.197301 5165 leader_election.cc:334] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.511s (SENT)
I20260929 14:12:03.204196 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:03.205662 5274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1439) took 2435 ms. Trace:
I20260929 14:12:03.178860 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:03.161365 7149 consensus_queue.cc:1035] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
W20260929 14:12:03.207888 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 67966.4us
I20260929 14:12:03.221552 5269 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:50768 (request call id 921) took 2886 ms. Trace:
I20260929 14:12:03.217643 5274 rpcz_store.cc:276] 0929 14:12:00.770141 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:02.628718 (+1858577us) service_pool.cc:227] Handling call
0929 14:12:02.629185 (+ 467us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:02.758977 (+129792us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:02.759022 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:02.785503 (+ 26481us) log.cc:842] Serialized 26 byte log entry
0929 14:12:03.141401 (+355898us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:03.141451 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:03.141545 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:03.205449 (+ 63904us) raft_consensus.cc:1671] finished
0929 14:12:03.205472 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:03.205606 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":338812,"threads_started":1,"wal-append.queue_time_us":399261}
I20260929 14:12:03.221937 5269 rpcz_store.cc:276] 0929 14:12:00.321955 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:03.123824 (+2801869us) service_pool.cc:227] Handling call
0929 14:12:03.208744 (+ 84920us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:03.222760 5165 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.192826 7209 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:03.201815 6822 raft_consensus.cc:3032] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:12:03.201951 7041 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:03.230791 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:03.232813 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:03.238998 5275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1453) took 1008 ms. Trace:
I20260929 14:12:03.239503 5275 rpcz_store.cc:276] 0929 14:12:02.231036 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:02.935723 (+704687us) service_pool.cc:227] Handling call
0929 14:12:02.936064 (+ 341us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:03.099972 (+163908us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:03.235662 (+135690us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b54003b0954. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:03.238901 (+ 3239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":368146176}
I20260929 14:12:03.224929 7208 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:03.246269 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:03.244612 5348 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:12:03.229597 6917 raft_consensus.cc:3032] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:12:03.233743 7209 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:03.249476 5242 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.236332 7237 consensus_queue.cc:1046] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260929 14:12:03.231137 6822 raft_consensus.cc:729] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:03.238405 7041 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:03.213991 7195 consensus_queue.cc:1046] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260929 14:12:03.255854 5347 pending_rounds.cc:76] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4: Aborting all ops after (but not including) 5
I20260929 14:12:03.257560 5347 pending_rounds.cc:97] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260929 14:12:03.257822 5347 raft_consensus.cc:2866] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:12:03.227535 7239 consensus_queue.cc:1046] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260929 14:12:03.259539 5870 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:03.242342 7234 consensus_queue.cc:1046] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260929 14:12:03.269507 5349 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 158) took 1210 ms. Trace:
I20260929 14:12:03.250238 7002 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:03.257292 7058 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:12:03.250905 6917 raft_consensus.cc:729] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:03.279276 7189 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:03.280607 7208 raft_consensus.cc:2726] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:03.301621 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 115871us
I20260929 14:12:03.281716 6907 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:03.273859 6822 consensus_queue.cc:260] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:03.312698 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 142652us
I20260929 14:12:03.267334 7191 raft_consensus.cc:3037] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:03.269896 5349 rpcz_store.cc:276] 0929 14:12:02.054976 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:02.093348 (+ 38372us) service_pool.cc:227] Handling call
0929 14:12:02.107203 (+ 13855us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:02.147578 (+ 40375us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:02.147610 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:02.148335 (+ 725us) log.cc:842] Serialized 26 byte log entry
0929 14:12:02.873104 (+724769us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:12:02.873143 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:02.873203 (+ 60us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:03.265367 (+392164us) raft_consensus.cc:1671] finished
0929 14:12:03.265386 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:03.265552 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8158,"raft.queue_time_us":18976,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":114,"thread_start_us":724188,"threads_started":1,"wal-append.queue_time_us":913198}
W20260929 14:12:03.346665 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 98778.5us
I20260929 14:12:03.301301 7189 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:03.363250 7233 raft_consensus.cc:3032] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Stepping down as leader of term 3
W20260929 14:12:03.367508 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 128383us
I20260929 14:12:03.347476 6822 raft_consensus.cc:3037] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:12:03.393816 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 123573us
I20260929 14:12:03.346052 6907 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:03.355808 7189 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:12:03.410321 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 161177us
I20260929 14:12:03.342211 6912 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:03.416680 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 157204us
I20260929 14:12:03.412252 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:03.394394 7233 raft_consensus.cc:729] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:03.429473 5242 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.411099 7237 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:03.430476 5316 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:12:03.443768 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 50343.2us
I20260929 14:12:03.447649 7237 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:03.419291 7149 consensus_queue.cc:1035] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260929 14:12:03.414554 6907 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:03.432008 6868 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:03.419137 7205 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:03.457540 7265 consensus_queue.cc:1035] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.004s
I20260929 14:12:03.458518 5316 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:03.459545 7263 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:03.461757 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:03.495850 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:03.496006 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 47141.2us
I20260929 14:12:03.496909 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:03.468423 7002 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:03.459599 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 107072us
I20260929 14:12:03.466447 6917 consensus_queue.cc:260] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:03.495733 5267 raft_consensus.cc:2454] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:12:03.497613 5269 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
W20260929 14:12:03.505695 7253 log.cc:924] Time spent T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20260929 14:12:03.505954 7263 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:03.506764 7249 log.cc:924] Time spent T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.107s user 0.000s sys 0.001s
I20260929 14:12:03.508327 7263 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:03.508677 7248 log.cc:924] Time spent T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.197s user 0.001s sys 0.000s
I20260929 14:12:03.510452 6907 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:03.511206 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:03.511559 6562 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:03.503218 7002 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:12:03.523180 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 64830.9us
I20260929 14:12:03.525612 5273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1440) took 2751 ms. Trace:
I20260929 14:12:03.494885 7246 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:03.526108 5273 rpcz_store.cc:276] 0929 14:12:00.773819 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:02.763528 (+1989709us) service_pool.cc:227] Handling call
0929 14:12:02.797546 (+ 34018us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:02.849324 (+ 51778us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:02.849368 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:02.865456 (+ 16088us) log.cc:842] Serialized 26 byte log entry
0929 14:12:03.182474 (+317018us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:03.182509 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:03.182578 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:03.525384 (+342806us) raft_consensus.cc:1671] finished
0929 14:12:03.525404 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:03.525550 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":13117,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":4144,"thread_start_us":285697,"threads_started":1,"wal-append.queue_time_us":362017}
I20260929 14:12:03.496147 6717 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260929 14:12:03.522154 7262 log.cc:924] Time spent T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.078s user 0.001s sys 0.001s
I20260929 14:12:03.521198 6917 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:03.535681 5871 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:03.519308 7245 raft_consensus.cc:3032] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:12:03.513147 7237 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:03.499900 7233 consensus_queue.cc:260] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:03.544862 7205 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:03.504156 6868 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:03.545822 6912 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:03.546042 7263 raft_consensus.cc:2781] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:03.549199 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:03.534025 7199 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:03.502698 7040 consensus_queue.cc:1035] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.039s
I20260929 14:12:03.541380 7245 raft_consensus.cc:729] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:03.519747 7041 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:03.555785 5165 leader_election.cc:334] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.530s (SENT)
I20260929 14:12:03.458532 7256 consensus_queue.cc:1035] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.179s
I20260929 14:12:03.568838 7041 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:03.569586 5347 raft_consensus.cc:2354] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
W20260929 14:12:03.563194 7202 consensus_meta.cc:287] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.534s user 0.011s sys 0.002s
I20260929 14:12:03.574700 6868 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:03.566530 5349 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:12:03.559583 7263 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:03.578804 5870 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:03.579152 7263 raft_consensus.cc:3037] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
W20260929 14:12:03.580108 7226 log.cc:924] Time spent T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.690s user 0.005s sys 0.000s
I20260929 14:12:03.580195 7261 raft_consensus.cc:2781] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:03.580760 7261 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:03.581291 7261 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:03.585698 7233 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:03.586948 5276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1458) took 1354 ms. Trace:
I20260929 14:12:03.587343 5276 rpcz_store.cc:276] 0929 14:12:02.232811 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:02.965235 (+732424us) service_pool.cc:227] Handling call
0929 14:12:02.984371 (+ 19136us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:03.002167 (+ 17796us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:12:03.230874 (+228707us) log.cc:842] Serialized 14 byte log entry
0929 14:12:03.393047 (+162173us) log.cc:842] Serialized 14 byte log entry
0929 14:12:03.460983 (+ 67936us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b18001b7ea0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b45638 00007fbda1b48b11 00007fbda1b48ac9 00007fbda1b48a59 00007fbda1b48a21 00007fbda1b47d1d 00007fbdaada9204 00007fbdaada9039 00007fbda1b27dc6
0929 14:12:03.586487 (+125504us) spinlock_profiling.cc:243] Waited 3.03 ms on lock 0x7b18001b7ea0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1abc4ca 00007fbdab19df99 00007fbda1980990 00007fbda1b42864 00007fbda1b42155 00007fbda4a653d0 00007fbda4a77576 00007fbda4a880d7 00007fbda4a88058
0929 14:12:03.586642 (+ 155us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:03.586684 (+ 42us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:03.586732 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:03.586799 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:03.586900 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1644,"prepare.queue_time_us":291673,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":75,"raft.queue_time_us":23499,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":10578,"spinlock_wait_cycles":14102144,"thread_start_us":342483,"threads_started":2,"wal-append.queue_time_us":172408}
I20260929 14:12:03.589366 7239 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:03.589956 7239 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:03.590456 7245 consensus_queue.cc:260] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:03.591771 7245 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:03.592238 7239 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:03.593526 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 173974us
I20260929 14:12:03.564632 7199 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:03.594285 5242 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.595366 5242 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.595763 7199 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:03.596470 5242 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.597774 5165 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.597201 7237 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:03.600070 7237 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:03.602591 7237 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:03.605962 6587 raft_consensus.cc:3032] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:03.606389 6587 raft_consensus.cc:729] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:03.607155 6587 consensus_queue.cc:260] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:03.612397 6912 raft_consensus.cc:2781] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:12:03.612813 6912 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:03.613206 6912 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:03.614168 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:03.515621 7209 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:03.616276 7209 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:03.616642 7219 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:12:03.617417 6946 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:03.617865 6946 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:03.618240 6946 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:03.552078 5316 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:03.618985 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:03.619508 6868 raft_consensus.cc:2726] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:03.619696 5271 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
W20260929 14:12:03.619692 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 67698.4us
I20260929 14:12:03.620458 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:03.620882 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:03.621625 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:03.622098 6991 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:03.622530 6991 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:12:03.557123 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 57697.1us
I20260929 14:12:03.545866 7002 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:03.612229 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:03.621645 5316 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:03.625262 5871 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
W20260929 14:12:03.625517 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 227410us
W20260929 14:12:03.552588 7264 log.cc:924] Time spent T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.101s user 0.001s sys 0.000s
I20260929 14:12:03.636143 7198 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:03.636175 7208 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:03.641571 7208 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:03.642474 7002 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:03.642802 7209 raft_consensus.cc:2781] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:12:03.643210 7209 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:03.643568 7209 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:03.643781 5239 leader_election.cc:398] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:03.644073 5239 leader_election.cc:401] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260929 14:12:03.644434 7208 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:03.646333 5314 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:03.648612 5871 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.651984 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:03.653373 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:03.647693 7040 raft_consensus.cc:2781] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
W20260929 14:12:03.658646 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 102892us
I20260929 14:12:03.658981 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:12:03.659145 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 62751.5us
W20260929 14:12:03.659983 5165 leader_election.cc:334] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.576s (SENT)
W20260929 14:12:03.660898 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 56189.6us
I20260929 14:12:03.661170 5242 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.661453 5870 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.664178 5165 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:03.679425 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 57884.7us
I20260929 14:12:03.664014 7198 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:03.665341 7233 raft_consensus.cc:2726] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:03.670096 6587 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:03.688264 5242 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.693596 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:03.695466 7267 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:03.697814 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:03.679633 7040 raft_consensus.cc:686] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:03.698602 5900 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:03.699363 7040 consensus_queue.cc:237] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:03.701946 5165 leader_election.cc:334] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.611s (SENT)
I20260929 14:12:03.702626 5165 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:12:03.702152 7258 log.cc:924] Time spent T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.194s user 0.002s sys 0.000s
I20260929 14:12:03.703743 5316 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:03.672392 7256 raft_consensus.cc:2726] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:12:03.681317 7208 raft_consensus.cc:2726] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:03.686745 7234 raft_consensus.cc:3032] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:12:03.708197 7219 raft_consensus.cc:2726] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:03.720444 7234 raft_consensus.cc:729] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
W20260929 14:12:03.715047 7252 log.cc:924] Time spent T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.296s user 0.000s sys 0.002s
I20260929 14:12:03.721242 5273 raft_consensus.cc:1261] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 14:12:03.721417 7234 consensus_queue.cc:260] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:03.717384 7277 consensus_queue.cc:1046] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260929 14:12:03.722725 7234 raft_consensus.cc:3037] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:03.722947 5316 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:03.723313 7277 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:03.722982 7235 consensus_queue.cc:1035] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:12:03.723708 7277 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:03.724126 7277 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:03.724350 6868 raft_consensus.cc:3032] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:12:03.724538 7259 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:03.724768 6868 raft_consensus.cc:729] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:03.725068 7259 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:03.725529 7259 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:03.704046 5343 raft_consensus.cc:1226] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:03.725744 6868 consensus_queue.cc:260] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:03.678541 7193 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260929 14:12:03.726917 6868 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:12:03.727439 6914 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:03.727937 6914 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:03.729997 6914 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:03.733974 6912 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:03.737434 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:12:03.741677 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 73942us
W20260929 14:12:03.741940 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 71632.1us
I20260929 14:12:03.751042 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:03.787951 5276 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
W20260929 14:12:03.808815 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 126804us
I20260929 14:12:03.829495 6642 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:03.830439 7271 consensus_queue.cc:1046] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.145s
I20260929 14:12:03.831707 7271 raft_consensus.cc:3032] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:12:03.832118 7271 raft_consensus.cc:729] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:03.833742 7273 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:03.741254 6912 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:03.849637 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 64425.1us
W20260929 14:12:03.916102 5167 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.3.242.68:35497 timed out after 1.719s (SENT) [suppressed 50 similar messages]
I20260929 14:12:03.956558 7268 consensus_queue.cc:1046] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:12:03.957651 7274 raft_consensus.cc:3032] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:03.958070 7274 raft_consensus.cc:729] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:03.908744 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:03.959404 5167 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
W20260929 14:12:03.960213 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 258301us
W20260929 14:12:03.960951 5165 leader_election.cc:334] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.620s (SENT)
W20260929 14:12:03.963429 5165 leader_election.cc:334] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.641s (SENT)
I20260929 14:12:03.963855 5165 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.964231 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:03.965605 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:03.947366 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:03.969935 5900 raft_consensus.cc:2454] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:12:03.954823 5242 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.975349 7058 raft_consensus.cc:2726] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:03.960287 7273 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:03.963266 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:03.952541 6946 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:03.959054 7274 consensus_queue.cc:260] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:03.996716 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:03.997630 7274 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:03.998117 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:03.998322 6946 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:03.998741 5906 raft_consensus.cc:2454] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 1.
I20260929 14:12:03.999622 5167 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:04.003561 5167 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:04.005017 5167 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:04.009634 5272 raft_consensus.cc:1226] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: []
W20260929 14:12:04.012552 5165 leader_election.cc:334] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.711s (SENT)
W20260929 14:12:04.012310 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 197514us
I20260929 14:12:04.015264 5167 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:04.009155 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 104976us
I20260929 14:12:04.018775 5165 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:03.866789 7152 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:04.013666 5276 raft_consensus.cc:1226] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
W20260929 14:12:04.021700 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 66979.6us
I20260929 14:12:04.021732 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:04.023530 7041 consensus_queue.cc:1035] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:04.024559 7267 raft_consensus.cc:2781] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:12:04.025002 7267 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:04.025420 7267 raft_consensus.cc:3037] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:12:04.001439 7273 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 14:12:04.014427 7279 log.cc:924] Time spent T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.135s user 0.001s sys 0.000s
I20260929 14:12:03.902545 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:04.018596 7268 raft_consensus.cc:2726] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:04.054662 7273 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:04.002059 7239 raft_consensus.cc:2781] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:04.055778 7239 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:04.056212 7239 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:04.021451 7058 raft_consensus.cc:2726] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:04.058405 5098 catalog_manager.cc:5495] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 3 to 5, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 5 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:04.060205 5098 catalog_manager.cc:5495] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:04.028298 5314 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:04.052680 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:04.057403 5268 raft_consensus.cc:1226] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:04.071091 5267 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:04.020154 6916 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260929 14:12:04.028752 7189 consensus_meta.cc:287] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.608s user 0.007s sys 0.003s
W20260929 14:12:04.073412 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 59157.1us
I20260929 14:12:04.072952 5098 catalog_manager.cc:5495] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:04.083961 6868 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260929 14:12:04.084177 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 55811.8us
I20260929 14:12:04.089679 7270 raft_consensus.cc:2726] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260929 14:12:04.090898 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 110363us
W20260929 14:12:04.091490 5165 leader_election.cc:334] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.735s (SENT)
I20260929 14:12:04.099695 5242 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:04.101066 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:04.101195 5165 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:04.108891 5098 catalog_manager.cc:5495] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:04.084421 6868 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:04.024590 6991 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:04.116590 5098 catalog_manager.cc:5495] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 3 to 4, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 4 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:04.139945 5242 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:04.141824 5098 catalog_manager.cc:5495] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:04.143785 6562 consensus_queue.cc:1035] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:12:04.145327 7208 raft_consensus.cc:2781] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:12:04.145836 7208 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:04.146286 7208 raft_consensus.cc:3037] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:04.091782 7189 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:04.144531 5098 catalog_manager.cc:5495] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:04.142624 5316 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:04.150967 5098 catalog_manager.cc:5495] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:04.153450 5098 catalog_manager.cc:5495] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:04.154709 5347 raft_consensus.cc:1226] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:04.149576 5097 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44250 (request call id 125) took 8199 ms. Trace:
I20260929 14:12:03.833104 7271 consensus_queue.cc:260] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:04.154515 5098 catalog_manager.cc:5495] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 3 to 4, leader changed from 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 4 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260929 14:12:04.027942 7277 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:04.161664 5098 catalog_manager.cc:5495] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 2 to 4, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 4 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:04.046432 7152 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:04.138579 6868 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
W20260929 14:12:04.167157 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 47434.4us
I20260929 14:12:04.167690 5098 catalog_manager.cc:5495] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:04.098570 7259 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:04.164088 7263 consensus_meta.cc:287] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.584s user 0.002s sys 0.009s
I20260929 14:12:04.171794 5098 catalog_manager.cc:5495] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:04.181320 5098 catalog_manager.cc:5495] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 2 to 6, leader changed from 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 6 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260929 14:12:04.137986 7040 consensus_queue.cc:1035] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.055s
I20260929 14:12:04.165740 6822 raft_consensus.cc:2781] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:04.185055 5098 catalog_manager.cc:5495] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:04.157698 7289 raft_consensus.cc:2726] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:04.167784 6642 consensus_queue.cc:1046] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.026s
I20260929 14:12:04.193845 6642 raft_consensus.cc:3032] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:04.194147 6642 raft_consensus.cc:729] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:04.194903 6642 consensus_queue.cc:260] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:04.196064 6642 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:04.163048 7219 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:04.197324 7219 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:04.188437 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 96965.3us
I20260929 14:12:04.198179 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:04.198460 7208 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:04.200726 7208 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:04.190217 5098 catalog_manager.cc:5495] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 4, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
W20260929 14:12:04.198724 5165 leader_election.cc:334] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.741s (SENT)
I20260929 14:12:04.201797 5165 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:04.204319 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:04.204718 5200 raft_consensus.cc:1261] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260929 14:12:04.206184 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:04.207222 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:04.165293 6916 raft_consensus.cc:2726] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:04.208086 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:04.211916 5242 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:04.207302 6991 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:04.240592 7261 consensus_meta.cc:287] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.659s user 0.011s sys 0.000s
I20260929 14:12:04.182920 7189 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:04.207537 7283 log.cc:924] Time spent T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260929 14:12:04.241801 7277 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:04.191967 6822 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:04.241922 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:04.178839 7041 consensus_queue.cc:1046] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260929 14:12:04.173709 7256 consensus_queue.cc:1035] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
W20260929 14:12:04.240306 7284 log.cc:924] Time spent T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.133s user 0.000s sys 0.000s
I20260929 14:12:04.243520 5268 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:04.185794 7263 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:04.245106 5903 raft_consensus.cc:1226] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:04.246084 7263 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:04.242604 6822 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:04.251024 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:12:04.253054 7285 log.cc:924] Time spent T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260929 14:12:04.253942 5898 raft_consensus.cc:1226] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 6. Current term is 7. Ops: []
W20260929 14:12:04.254338 7282 log.cc:924] Time spent T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.151s user 0.002s sys 0.000s
W20260929 14:12:04.247401 7209 consensus_meta.cc:287] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.603s user 0.006s sys 0.005s
I20260929 14:12:04.255060 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:04.269790 5269 raft_consensus.cc:3037] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:04.236040 7205 raft_consensus.cc:2726] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:04.259465 5199 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:12:04.260082 5194 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:12:04.156265 5097 rpcz_store.cc:276] 0929 14:11:55.911694 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:11:55.911863 (+ 169us) service_pool.cc:227] Handling call
0929 14:12:04.111468 (+8199605us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:12:02.655465 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:12:02.656296 (+ 831us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:12:02.656350 (+ 54us) write_op.cc:435] Acquired schema lock
0929 14:12:02.656386 (+ 36us) tablet.cc:660] Decoding operations
0929 14:12:02.657362 (+ 976us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:12:02.657449 (+ 87us) write_op.cc:641] Partition lock acquired for write op
0929 14:12:02.657505 (+ 56us) tablet.cc:683] Acquiring locks for 6 operations
0929 14:12:02.658254 (+ 749us) tablet.cc:699] Row locks acquired
0929 14:12:02.658283 (+ 29us) write_op.cc:260] PREPARE: finished
0929 14:12:02.658476 (+ 193us) write_op.cc:270] Start()
0929 14:12:02.658611 (+ 135us) write_op.cc:276] Timestamp: P: 1790691122658404 usec, L: 0
0929 14:12:02.658645 (+ 34us) op_driver.cc:331] REPLICATION: starting
0929 14:12:02.659425 (+ 780us) log.cc:842] Serialized 2430 byte log entry
0929 14:12:03.396481 (+737056us) op_driver.cc:444] REPLICATION: finished
0929 14:12:03.638707 (+242226us) write_op.cc:301] APPLY: starting
0929 14:12:03.638834 (+ 127us) tablet.cc:1365] starting BulkCheckPresence
0929 14:12:03.639100 (+ 266us) tablet.cc:1368] finished BulkCheckPresence
0929 14:12:03.639240 (+ 140us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:12:03.969848 (+330608us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:12:03.970199 (+ 351us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0929 14:12:03.970263 (+ 64us) write_op.cc:312] APPLY: finished
0929 14:12:03.970640 (+ 377us) log.cc:842] Serialized 53 byte log entry
0929 14:12:03.970847 (+ 207us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:03.970972 (+ 125us) write_op.cc:454] Released schema lock
0929 14:12:03.971061 (+ 89us) write_op.cc:341] FINISH: Updating metrics
0929 14:12:03.971374 (+ 313us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:03.971401 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":42614,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":218468,"apply.run_cpu_time_us":4697,"apply.run_wall_time_us":333384,"num_ops":6,"prepare.queue_time_us":617555,"prepare.run_cpu_time_us":16389,"prepare.run_wall_time_us":434089,"raft.queue_time_us":251535,"raft.run_cpu_time_us":10192,"raft.run_wall_time_us":217429,"replication_time_us":709637,"thread_start_us":779896,"threads_started":4,"wal-append.queue_time_us":428060}]]}
I20260929 14:12:04.262727 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:04.263415 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:04.264436 7288 log.cc:924] Time spent T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
I20260929 14:12:04.266266 7152 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:04.266494 7280 log.cc:924] Time spent T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.112s user 0.001s sys 0.000s
I20260929 14:12:04.267767 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:04.285805 5342 raft_consensus.cc:2454] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:12:04.239672 5347 raft_consensus.cc:2379] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 5: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:04.288743 5274 raft_consensus.cc:1226] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:04.238904 5098 catalog_manager.cc:5495] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:04.278671 5199 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:04.284097 5267 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
W20260929 14:12:04.269374 7198 consensus_meta.cc:287] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.579s user 0.007s sys 0.003s
I20260929 14:12:04.297287 5199 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:12:04.276137 7219 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:04.299134 5870 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:04.241029 7261 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:04.287186 5194 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:04.306140 5870 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:04.287441 6587 raft_consensus.cc:2726] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:04.305646 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:04.269167 7276 consensus_queue.cc:1046] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260929 14:12:04.302897 5098 catalog_manager.cc:5495] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 4, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:04.309578 5275 raft_consensus.cc:2454] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:12:04.313308 5098 catalog_manager.cc:5495] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 3, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 3 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:04.314870 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:04.331746 6562 consensus_queue.cc:1046] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:12:04.333108 7297 raft_consensus.cc:3032] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:12:04.333619 7297 raft_consensus.cc:729] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:04.333731 7259 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:04.334761 7297 consensus_queue.cc:260] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:04.336618 7297 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:04.336788 5199 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:04.297904 7198 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:04.339769 5314 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:04.282389 7209 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:04.308959 5194 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:04.346444 5199 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:04.327543 5098 catalog_manager.cc:5495] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 2, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 2 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
W20260929 14:12:04.342772 5165 leader_election.cc:334] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.843s (SENT)
I20260929 14:12:04.308669 7152 raft_consensus.cc:2781] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:04.360910 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:12:04.361685 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 173572us
I20260929 14:12:04.362382 5192 raft_consensus.cc:1261] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260929 14:12:04.408226 7234 consensus_queue.cc:1035] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260929 14:12:04.408627 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:04.409792 7259 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:04.409993 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:04.410629 5192 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:12:04.410372 7259 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:04.411644 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:04.412294 5192 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
W20260929 14:12:04.480397 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 137595us
I20260929 14:12:04.550951 5270 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:04.558920 5873 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:04.558207 5098 catalog_manager.cc:5495] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 2 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
W20260929 14:12:04.535152 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 159134us
I20260929 14:12:04.364038 5269 raft_consensus.cc:2454] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:12:04.562764 5195 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:04.315725 7121 raft_consensus.cc:2726] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:12:04.415400 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:12:04.568885 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 53701.6us
I20260929 14:12:04.568594 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
W20260929 14:12:04.564949 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 220961us
I20260929 14:12:04.510919 5904 raft_consensus.cc:1226] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5. Ops: []
W20260929 14:12:04.484688 7288 log.cc:924] Time spent T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.113s user 0.000s sys 0.000s
I20260929 14:12:04.560249 5098 catalog_manager.cc:5495] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 2, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 2 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:04.580255 5269 raft_consensus.cc:1226] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:04.565910 5165 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:04.464340 7254 log.cc:924] Time spent T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.119s user 0.000s sys 0.000s
I20260929 14:12:04.583200 5903 raft_consensus.cc:2379] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 5: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:12:04.412503 7259 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:04.412796 7269 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:04.559079 7152 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:04.592810 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:04.593470 5242 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:04.519126 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:04.591416 6991 consensus_queue.cc:1046] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260929 14:12:04.594295 7202 raft_consensus.cc:2726] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:04.599406 7234 raft_consensus.cc:2726] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260929 14:12:04.599536 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:04.617998 5201 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:04.344125 7267 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:04.590394 5098 catalog_manager.cc:5495] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 2, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 2 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
W20260929 14:12:04.610054 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 49793.5us
I20260929 14:12:04.611573 5871 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
W20260929 14:12:04.601063 7239 consensus_meta.cc:287] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.544s user 0.007s sys 0.000s
W20260929 14:12:04.619431 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 276681us
W20260929 14:12:04.621137 5165 leader_election.cc:334] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.876s (SENT)
I20260929 14:12:04.621742 5165 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:04.353605 7273 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:04.628044 5098 catalog_manager.cc:5495] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
W20260929 14:12:04.651391 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 66039.6us
I20260929 14:12:04.618362 7208 raft_consensus.cc:2726] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:04.651335 7234 raft_consensus.cc:2781] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:12:04.610414 7152 raft_consensus.cc:3037] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:04.653537 7269 raft_consensus.cc:686] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:04.678635 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:04.686578 5197 raft_consensus.cc:2379] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 7: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:04.688640 7274 raft_consensus.cc:3032] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:12:04.689157 7274 raft_consensus.cc:729] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:04.690200 7274 consensus_queue.cc:260] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:04.631377 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:04.683759 7261 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:04.626626 7239 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:04.733951 7234 raft_consensus.cc:686] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [term 4 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:04.736479 7209 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:04.654754 6868 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:04.753638 7121 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:04.683693 7302 consensus_queue.cc:1046] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.125s
W20260929 14:12:04.799054 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 115639us
I20260929 14:12:04.790012 5271 raft_consensus.cc:1226] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:04.797901 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:04.800068 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 143327us
I20260929 14:12:04.800756 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:12:04.803741 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 244930us
I20260929 14:12:04.803968 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:04.772092 7208 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:04.817200 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 196088us
W20260929 14:12:04.818432 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 243909us
I20260929 14:12:04.820514 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:04.828353 5165 leader_election.cc:334] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.880s (SENT)
I20260929 14:12:04.828745 5165 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:04.827129 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:04.841524 7040 raft_consensus.cc:3032] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:04.841971 7040 raft_consensus.cc:729] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:04.842769 7040 consensus_queue.cc:260] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:04.850389 5270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50776 (request call id 1485) took 1027 ms. Trace:
I20260929 14:12:04.850788 5270 rpcz_store.cc:276] 0929 14:12:03.822785 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:04.010179 (+187394us) service_pool.cc:227] Handling call
0929 14:12:04.850218 (+840039us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b54003bf954. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:04.850329 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":749033728}
I20260929 14:12:04.860194 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:04.822496 7274 raft_consensus.cc:3037] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:04.865548 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:04.866430 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:04.869803 5274 raft_consensus.cc:2379] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 4: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:04.797546 5194 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:04.864975 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:04.875617 5347 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:12:04.837069 7235 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:04.797420 7121 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:04.837323 7276 raft_consensus.cc:2726] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:04.871537 5200 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:04.878741 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:04.884420 5199 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:04.883484 5164 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:04.822679 7208 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:04.899569 7208 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:04.889523 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:04.901855 7121 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:04.867249 7198 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260929 14:12:04.868642 7273 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:04.891251 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:04.875284 7239 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:04.895637 5268 consensus_meta.cc:287] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.650s user 0.007s sys 0.001s
I20260929 14:12:04.906729 5201 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:04.935966 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:04.940850 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:04.900990 5898 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:04.938652 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:04.877661 7267 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:04.939229 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 44470.6us
W20260929 14:12:04.945088 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 125584us
I20260929 14:12:04.942178 5346 raft_consensus.cc:2454] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:12:04.953537 5870 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:04.957321 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:04.957476 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:04.958119 5275 raft_consensus.cc:2454] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
W20260929 14:12:04.958475 7305 log.cc:924] Time spent T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20260929 14:12:04.959299 5167 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:04.959393 5871 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:04.961166 7261 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:12:04.961373 7273 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260929 14:12:04.966120 5165 leader_election.cc:334] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.900s (SENT)
I20260929 14:12:04.966683 5165 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:04.969542 5198 raft_consensus.cc:1226] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260929 14:12:04.940101 7276 raft_consensus.cc:2781] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader election won for term 2
I20260929 14:12:04.938941 7268 consensus_queue.cc:1046] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260929 14:12:04.871268 7234 consensus_queue.cc:237] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:04.857355 7269 consensus_queue.cc:237] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:04.889721 7235 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:04.991752 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:04.961767 7261 raft_consensus.cc:2781] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:04.947912 5268 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:04.997691 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:05.008328 5342 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:05.009819 5242 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:05.019164 7271 raft_consensus.cc:3037] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:04.995796 7267 raft_consensus.cc:2726] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:05.003331 7261 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 14:12:05.032563 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 81713.6us
W20260929 14:12:05.037205 5165 leader_election.cc:334] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 2.008s (SENT)
I20260929 14:12:05.032559 5239 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:05.007439 6868 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:04.983800 7239 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.040s
I20260929 14:12:05.047727 5195 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
W20260929 14:12:05.036506 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 75731.9us
W20260929 14:12:05.064786 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 103843us
I20260929 14:12:05.038183 7261 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:12:05.045701 6822 consensus_meta.cc:287] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.797s user 0.012s sys 0.000s
I20260929 14:12:05.059626 5314 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:05.041342 7246 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:05.093327 5274 raft_consensus.cc:1226] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:05.037983 7317 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:05.096172 7246 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.097710 7310 consensus_queue.cc:1046] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:12:05.098507 7246 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:05.099488 7246 raft_consensus.cc:2781] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:12:05.100350 7246 raft_consensus.cc:686] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
W20260929 14:12:05.101341 7297 consensus_meta.cc:287] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.761s user 0.012s sys 0.000s
W20260929 14:12:05.102187 7307 log.cc:924] Time spent T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.148s user 0.000s sys 0.001s
I20260929 14:12:05.101534 7246 consensus_queue.cc:237] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.052294 7202 raft_consensus.cc:2726] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:05.059495 7198 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260929 14:12:05.104698 5165 leader_election.cc:334] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 2.039s (SENT)
I20260929 14:12:05.065363 5242 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:05.096031 6822 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.058763 6868 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:05.105484 5194 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:05.112486 5906 raft_consensus.cc:1226] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:05.114665 7286 consensus_queue.cc:1035] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:05.115087 5316 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:05.072743 6917 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:05.116986 5192 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:05.118497 5165 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:05.171439 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.174058 5275 raft_consensus.cc:2454] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:12:05.174980 5871 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:05.112994 7297 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:05.178153 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:05.177646 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 62625.8us
I20260929 14:12:05.182245 5316 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:05.182440 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:05.118240 7198 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:05.183105 5192 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:12:05.183971 5342 raft_consensus.cc:2454] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
W20260929 14:12:05.186664 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 126122us
W20260929 14:12:05.182657 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 130796us
I20260929 14:12:05.170933 7276 raft_consensus.cc:686] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:05.175199 7289 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260929 14:12:05.190163 5165 leader_election.cc:334] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 2.070s (SENT)
W20260929 14:12:05.191516 7304 log.cc:924] Time spent T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.568s user 0.000s sys 0.006s
I20260929 14:12:05.188565 5316 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:05.198117 5903 raft_consensus.cc:1226] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:05.184454 7269 raft_consensus.cc:2726] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:05.220549 5200 raft_consensus.cc:2396] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260929 14:12:05.188926 7198 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:05.184600 7235 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:05.184716 7325 raft_consensus.cc:2781] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:05.136497 6868 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.139768 7152 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.222710 5165 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:05.226756 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:05.234814 5276 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:05.227248 7259 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:05.185140 7328 raft_consensus.cc:2726] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:12:05.177183 6917 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.258926 6917 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:05.235478 7002 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260929 14:12:05.228938 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:05.264372 6717 raft_consensus.cc:3032] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:05.264775 6717 raft_consensus.cc:729] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:05.264844 5194 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:05.265339 5898 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
W20260929 14:12:05.273478 5193 consensus_meta.cc:287] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.943s user 0.006s sys 0.006s
I20260929 14:12:05.273870 5193 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:05.274416 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52182 (request call id 1477) took 1017 ms. Trace:
I20260929 14:12:05.274735 5193 rpcz_store.cc:276] 0929 14:12:04.272465 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:04.285279 (+ 12814us) service_pool.cc:227] Handling call
0929 14:12:05.274357 (+989078us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260929 14:12:05.277561 7315 log.cc:924] Time spent T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.159s user 0.001s sys 0.000s
I20260929 14:12:05.282879 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:05.283783 5267 raft_consensus.cc:3037] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 5
I20260929 14:12:05.285853 7205 raft_consensus.cc:3032] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:05.286576 7205 raft_consensus.cc:729] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:05.287372 7303 raft_consensus.cc:3032] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:12:05.290596 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:05.291231 5342 raft_consensus.cc:3037] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:05.294077 5267 raft_consensus.cc:2454] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
W20260929 14:12:05.253791 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 66530.5us
W20260929 14:12:05.295739 5165 leader_election.cc:334] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 2.117s (SENT)
I20260929 14:12:05.186347 7297 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.229544 7325 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:05.315135 5871 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:05.320118 7261 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.287863 7303 raft_consensus.cc:729] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:05.238444 7267 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:05.343222 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:05.293075 7152 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:05.231549 7274 consensus_queue.cc:1046] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.104s
I20260929 14:12:05.343976 5907 raft_consensus.cc:2454] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
W20260929 14:12:05.370342 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 74636.9us
W20260929 14:12:05.373894 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 185441us
I20260929 14:12:05.335433 7325 raft_consensus.cc:3037] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:05.346918 7263 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:05.394065 7263 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.396299 7263 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:05.397346 6822 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:05.340819 5268 raft_consensus.cc:1261] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:12:05.343654 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:05.344524 7235 raft_consensus.cc:2781] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:05.392062 7297 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:05.392138 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:05.393543 7274 raft_consensus.cc:3032] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:05.402133 5316 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:05.401029 5194 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
W20260929 14:12:05.404110 5165 leader_election.cc:334] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 2.122s (SENT)
I20260929 14:12:05.427026 5165 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:05.416457 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1495) took 1536 ms. Trace:
I20260929 14:12:05.427647 5271 rpcz_store.cc:276] 0929 14:12:03.880450 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:04.259338 (+378888us) service_pool.cc:227] Handling call
0929 14:12:04.259741 (+ 403us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:04.799887 (+540146us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:05.416143 (+616256us) spinlock_profiling.cc:243] Waited 526 ms on lock 0x7b5400392e54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:05.416374 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1210409728}
W20260929 14:12:05.416988 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 166878us
I20260929 14:12:05.417912 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:12:05.419602 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 304109us
I20260929 14:12:05.420502 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.287441 7205 consensus_queue.cc:260] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.422276 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:05.430831 5167 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:12:05.424103 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:05.377121 7286 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:05.433249 5197 raft_consensus.cc:1261] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:12:05.442094 6868 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:05.414767 7289 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:05.426363 7302 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:05.462227 5314 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:05.461087 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:05.426149 7274 raft_consensus.cc:729] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:05.431887 7235 raft_consensus.cc:686] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [term 5 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:05.442884 7202 raft_consensus.cc:2726] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:05.479950 5201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 196) took 1146 ms. Trace:
I20260929 14:12:05.465456 7289 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:05.484055 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 82071.8us
I20260929 14:12:05.470191 7302 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.388441 6717 consensus_queue.cc:260] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.478022 7274 consensus_queue.cc:260] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:05.487128 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 110728us
W20260929 14:12:05.493510 5165 leader_election.cc:334] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 2.155s (SENT)
I20260929 14:12:05.488993 7289 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:05.490300 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:05.490770 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:05.458020 7326 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:05.480403 5201 rpcz_store.cc:276] 0929 14:12:04.333778 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:04.947055 (+613277us) service_pool.cc:227] Handling call
0929 14:12:04.947419 (+ 364us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:04.984331 (+ 36912us) raft_consensus.cc:1520] Early marking committed up to index 7
0929 14:12:04.990822 (+ 6491us) log.cc:842] Serialized 14 byte log entry
0929 14:12:05.415483 (+424661us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:05.435775 (+ 20292us) log.cc:842] Serialized 26 byte log entry
0929 14:12:05.435904 (+ 129us) raft_consensus.cc:1631] Marking committed up to 8
0929 14:12:05.455449 (+ 19545us) log.cc:842] Serialized 14 byte log entry
0929 14:12:05.455636 (+ 187us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:05.455711 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:05.479772 (+ 24061us) raft_consensus.cc:1671] finished
0929 14:12:05.479794 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:05.479897 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":73413,"prepare.queue_time_us":26072,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":60306,"raft.run_cpu_time_us":484,"raft.run_wall_time_us":22033,"thread_start_us":394877,"threads_started":1,"wal-append.queue_time_us":424687}
I20260929 14:12:05.485721 5276 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:05.507344 5165 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:05.511155 5316 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:05.444901 7286 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.500146 7312 raft_consensus.cc:2781] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:05.501837 7335 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:05.511924 6717 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:05.513391 5268 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:05.526497 7259 raft_consensus.cc:2781] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:05.423192 7303 consensus_queue.cc:260] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.551198 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:05.555516 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:05.556118 5345 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 5
I20260929 14:12:05.556525 5906 raft_consensus.cc:2379] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 3: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:12:05.535617 7328 raft_consensus.cc:2726] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:12:05.537834 7261 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:05.538947 7289 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:05.538471 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:05.552217 7274 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:05.551779 7259 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:05.574656 7335 raft_consensus.cc:686] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:05.374900 7276 consensus_queue.cc:237] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.601339 7202 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:05.606209 7202 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.606568 7302 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:05.608186 7202 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:05.610087 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:05.612128 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:05.612973 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.613718 5273 raft_consensus.cc:2454] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:12:05.614024 7335 consensus_queue.cc:237] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:05.615070 7271 consensus_meta.cc:287] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.585s user 0.007s sys 0.004s
W20260929 14:12:05.599094 7331 log.cc:924] Time spent T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.224s user 0.000s sys 0.001s
I20260929 14:12:05.591106 7289 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.618225 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:05.618808 7289 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:05.594398 7326 raft_consensus.cc:686] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:05.620572 7326 consensus_queue.cc:237] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.623458 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:05.624284 5903 raft_consensus.cc:2354] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:05.624859 7268 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:05.625415 7268 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.550806 7235 consensus_queue.cc:237] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.630511 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:05.631490 5196 raft_consensus.cc:2454] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:12:05.632809 5871 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:05.637157 5167 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:05.641260 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 147742us
I20260929 14:12:05.595364 7259 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:05.635421 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:05.652112 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:05.653053 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:05.654261 5165 leader_election.cc:334] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 2.175s (SENT)
W20260929 14:12:05.654335 5873 leader_election.cc:334] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.063s (SENT)
I20260929 14:12:05.654963 5165 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:05.657557 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:05.657924 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:05.658802 5902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:05.658841 7289 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:05.660416 5904 raft_consensus.cc:2454] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:05.660282 7289 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.662484 7289 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:05.663857 5349 raft_consensus.cc:1226] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:05.666064 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:05.666590 5905 raft_consensus.cc:1261] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260929 14:12:05.666132 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:05.667528 5274 raft_consensus.cc:1226] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:05.667877 5267 raft_consensus.cc:2354] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5.
I20260929 14:12:05.655932 7273 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:05.666687 5276 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:05.668828 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.670830 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.672526 5274 raft_consensus.cc:1226] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:05.673552 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.673851 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:05.675844 6946 consensus_queue.cc:1046] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260929 14:12:05.677158 6946 raft_consensus.cc:3032] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:05.677590 6946 raft_consensus.cc:729] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:05.678063 7286 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:05.679004 6946 consensus_queue.cc:260] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.680961 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:05.693778 5276 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:12:05.692313 5273 raft_consensus.cc:2354] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:12:05.603099 7312 raft_consensus.cc:686] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:05.638826 7268 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:05.690939 5345 raft_consensus.cc:2454] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:12:05.723552 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:05.718003 5268 raft_consensus.cc:2454] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:12:05.720075 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.721338 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:05.722443 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.651803 7189 raft_consensus.cc:2781] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:05.727747 5871 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:05.731142 5199 raft_consensus.cc:1261] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 14:12:05.733798 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.720723 5900 raft_consensus.cc:2354] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5.
I20260929 14:12:05.741021 5268 raft_consensus.cc:2354] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:12:05.741029 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:37270 (request call id 190) took 1527 ms. Trace:
I20260929 14:12:05.731513 5349 raft_consensus.cc:2354] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:05.741511 5196 rpcz_store.cc:276] 0929 14:12:04.203382 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:04.205725 (+ 2343us) service_pool.cc:227] Handling call
0929 14:12:05.730547 (+1524822us) spinlock_profiling.cc:243] Waited 1.42 s on lock 0x7b540031e054. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:05.730705 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3271115520}
I20260929 14:12:05.751273 5903 raft_consensus.cc:1226] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7. Ops: [4.6-6.7]
I20260929 14:12:05.753381 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.754421 5268 raft_consensus.cc:2379] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:05.754969 5239 leader_election.cc:398] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:05.756081 5239 leader_election.cc:401] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
W20260929 14:12:05.757807 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 115666us
I20260929 14:12:05.728384 7189 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:05.759608 7208 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 4
I20260929 14:12:05.765682 7303 raft_consensus.cc:3037] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:05.758184 7209 raft_consensus.cc:2781] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:05.766414 7209 raft_consensus.cc:686] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 5 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:05.767251 5268 raft_consensus.cc:1261] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260929 14:12:05.767941 5165 leader_election.cc:334] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 2.236s (SENT)
I20260929 14:12:05.740260 7289 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:05.776479 5342 raft_consensus.cc:2454] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
W20260929 14:12:05.782464 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 236349us
I20260929 14:12:05.783382 5268 raft_consensus.cc:1226] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5. Ops: []
W20260929 14:12:05.785477 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 240625us
I20260929 14:12:05.785912 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.762042 6946 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:05.767666 7189 raft_consensus.cc:3037] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
W20260929 14:12:05.792438 5165 leader_election.cc:334] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 2.244s (SENT)
I20260929 14:12:05.796416 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:05.796216 5314 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:05.799772 5165 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:05.803126 5198 raft_consensus.cc:1226] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:12:05.784642 7273 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:05.808235 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.809348 5273 raft_consensus.cc:2379] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 5: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
W20260929 14:12:05.798342 7343 log.cc:924] Time spent T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20260929 14:12:05.778692 7263 consensus_queue.cc:1035] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260929 14:12:05.799050 7041 consensus_queue.cc:1046] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20260929 14:12:05.811124 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.823746 5267 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:05.802933 7341 consensus_queue.cc:1035] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.017s
I20260929 14:12:05.808763 7352 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:05.811686 5905 raft_consensus.cc:1226] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:05.825114 6562 consensus_queue.cc:1035] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:12:05.803735 6642 raft_consensus.cc:2726] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 14:12:05.806859 7297 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:05.784725 7289 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.831080 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.831419 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.831980 5273 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:12:05.834276 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:05.767251 7209 consensus_queue.cc:237] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.845937 7297 raft_consensus.cc:686] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:05.847095 7297 consensus_queue.cc:237] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.849335 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.850018 5271 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:05.863107 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.826205 7273 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.867925 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:05.871527 7273 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:05.793980 7312 consensus_queue.cc:237] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.874722 7259 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.875603 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:05.876689 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.877447 5270 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:05.881888 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.882962 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1493) took 1232 ms. Trace:
I20260929 14:12:05.883908 7289 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:05.858552 7325 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.883375 5193 rpcz_store.cc:276] 0929 14:12:04.651017 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:05.275587 (+624570us) service_pool.cc:227] Handling call
0929 14:12:05.275942 (+ 355us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:05.276342 (+ 400us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:05.276377 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:05.276837 (+ 460us) log.cc:842] Serialized 26 byte log entry
0929 14:12:05.516288 (+239451us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:05.519671 (+ 3383us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:05.527836 (+ 8165us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:05.882755 (+354919us) raft_consensus.cc:1671] finished
0929 14:12:05.882781 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:05.882904 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":7445,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"thread_start_us":232111,"threads_started":1,"wal-append.queue_time_us":324862}
I20260929 14:12:05.885946 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:05.886526 5898 raft_consensus.cc:2354] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:05.887101 7325 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:05.888688 7040 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:05.889364 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:05.889927 5349 raft_consensus.cc:3037] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:05.891064 7259 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:05.891886 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:05.892457 5274 raft_consensus.cc:3037] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:05.903236 5267 raft_consensus.cc:1226] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:05.904582 5316 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:05.906620 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 203) took 1570 ms. Trace:
I20260929 14:12:05.907011 5199 rpcz_store.cc:276] 0929 14:12:04.335715 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:05.015642 (+679927us) service_pool.cc:227] Handling call
0929 14:12:05.730719 (+715077us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:05.731672 (+ 953us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:05.906445 (+174773us) spinlock_profiling.cc:243] Waited 714 ms on lock 0x7b540031e050. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6ca09 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:05.906568 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1643023232}
I20260929 14:12:05.912267 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.912499 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:05.913316 5272 raft_consensus.cc:2354] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5.
I20260929 14:12:05.914053 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.914824 5271 raft_consensus.cc:2379] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:05.916447 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:05.917450 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.917619 7205 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:05.917550 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.918845 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.918951 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.919682 5272 raft_consensus.cc:2379] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 2: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:05.919971 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.920661 5270 raft_consensus.cc:2354] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5.
I20260929 14:12:05.921463 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:05.921895 7198 consensus_meta.cc:287] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.687s user 0.011s sys 0.000s
I20260929 14:12:05.922180 5267 raft_consensus.cc:2379] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:05.922273 7198 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.924358 7208 raft_consensus.cc:2726] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:05.929123 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:05.929981 5273 raft_consensus.cc:2354] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6.
I20260929 14:12:05.931195 7321 raft_consensus.cc:3032] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:05.931648 7321 raft_consensus.cc:729] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:05.924386 7198 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:05.931548 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:05.933847 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.934571 5198 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1480) took 1581 ms. Trace:
I20260929 14:12:05.934921 5198 rpcz_store.cc:276] 0929 14:12:04.352705 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:05.037774 (+685069us) service_pool.cc:227] Handling call
0929 14:12:05.051568 (+ 13794us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:05.803536 (+751968us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:05.934212 (+130676us) spinlock_profiling.cc:243] Waited 751 ms on lock 0x7b540031ef54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:05.934353 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1727951104}
I20260929 14:12:05.941457 5268 raft_consensus.cc:2354] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:12:05.944909 5275 raft_consensus.cc:2354] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6.
I20260929 14:12:05.944723 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.953163 5274 raft_consensus.cc:2454] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
W20260929 14:12:05.957888 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 102102us
I20260929 14:12:05.957957 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.958513 5871 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:05.960558 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.960784 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.961474 5272 raft_consensus.cc:2379] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:05.962101 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.965845 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:05.932631 7321 consensus_queue.cc:260] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.969202 7152 raft_consensus.cc:2781] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:12:05.969825 7152 raft_consensus.cc:686] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 4 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:05.970614 7152 consensus_queue.cc:237] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.972236 7302 consensus_queue.cc:1046] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:05.973189 7325 raft_consensus.cc:3032] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:12:05.976212 7325 raft_consensus.cc:729] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:05.976709 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:05.977337 5271 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:12:05.977710 5165 connection.cc:232] RPC call timeout handler was delayed by -1.5251627200000257s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1625s for a response
I20260929 14:12:05.977828 7189 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:05.978592 5165 connection.cc:232] RPC call timeout handler was delayed by -1.3148027880001791s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2303s for a response
W20260929 14:12:05.978930 5165 connection.cc:232] RPC call timeout handler was delayed by -1.3010627200001181s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1874s for a response
W20260929 14:12:05.979195 5165 connection.cc:232] RPC call timeout handler was delayed by -1.246339584999987s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1743s for a response
W20260929 14:12:05.979576 5165 connection.cc:232] RPC call timeout handler was delayed by -1.1662395850000848s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18320000000000003s for a response
I20260929 14:12:05.979940 7189 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:05.979849 5165 connection.cc:232] RPC call timeout handler was delayed by -1.0020670009998867s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1778s for a response
I20260929 14:12:05.981171 7267 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:05.974203 5273 raft_consensus.cc:1226] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:05.981664 7267 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.982527 5902 raft_consensus.cc:2354] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:05.982411 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:05.982019 5342 raft_consensus.cc:1226] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:05.982995 5273 raft_consensus.cc:3037] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:05.983893 7267 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:05.984424 7267 consensus_queue.cc:1046] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 14:12:05.985046 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:05.975406 5873 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:05.985374 7267 raft_consensus.cc:3032] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:05.985668 5342 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:12:05.985796 7267 raft_consensus.cc:729] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:05.986869 5164 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:05.960687 6642 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:05.986649 7267 consensus_queue.cc:260] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.986932 7263 raft_consensus.cc:2726] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:05.987768 7352 raft_consensus.cc:2781] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:12:05.988279 7352 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:05.988726 7352 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:05.987975 6642 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:05.988850 7263 consensus_queue.cc:1035] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:05.990733 6642 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:05.988358 5347 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 7.
I20260929 14:12:05.992084 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:05.997849 5870 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:05.998267 5346 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:05.998940 7209 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:05.999333 7209 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:05.976974 7325 consensus_queue.cc:260] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:05.999526 7321 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:05.999678 7209 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:06.000348 7325 raft_consensus.cc:3037] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:06.007886 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:06.009711 5314 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:06.010563 7352 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:06.011971 5349 raft_consensus.cc:2454] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:12:06.012329 5164 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:06.012975 5273 raft_consensus.cc:2454] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:12:06.015904 7353 raft_consensus.cc:2781] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:06.020532 5314 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:06.020948 7353 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 14:12:06.016641 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 112176us
I20260929 14:12:06.017109 7321 raft_consensus.cc:2726] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:12:06.019248 5271 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:06.022079 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:06.024950 5871 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:06.040820 7263 raft_consensus.cc:2781] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:12:06.037575 5907 raft_consensus.cc:2454] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:12:06.042747 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:06.043967 5268 raft_consensus.cc:2354] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5.
I20260929 14:12:06.051718 5316 leader_election.cc:398] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497) with higher term. Message: Invalid argument: T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5.
I20260929 14:12:06.052947 5316 leader_election.cc:401] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:12:06.052048 5314 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:06.059615 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:06.060284 5192 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:06.063093 7267 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:06.066274 5197 raft_consensus.cc:1261] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:12:06.069550 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:06.038139 7353 raft_consensus.cc:3037] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:06.067596 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:06.075770 5197 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 5
I20260929 14:12:06.076330 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:06.078401 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:06.078822 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:06.078850 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:06.078395 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:06.079059 5194 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:06.039359 7209 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:06.145675 5907 raft_consensus.cc:1226] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:06.045068 6717 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:06.147857 5201 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:12:06.045197 6642 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:06.148553 5195 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:06.056056 7263 raft_consensus.cc:686] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 4 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
W20260929 14:12:06.128446 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 52340.4us
I20260929 14:12:06.136047 5198 raft_consensus.cc:1226] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
W20260929 14:12:06.136976 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 65123.6us
I20260929 14:12:06.151259 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:06.070214 7318 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497) with higher term. Message: Invalid argument: T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5.
I20260929 14:12:06.149539 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:06.149394 5196 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:12:06.178912 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:06.079705 7311 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:06.182899 5271 raft_consensus.cc:2362] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4. Re-sending same reply.
I20260929 14:12:06.184552 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:06.075227 7352 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:06.146634 5902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:06.184702 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:06.166169 6717 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:06.314006 6717 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:06.258919 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:06.315506 5268 raft_consensus.cc:2379] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 5: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:06.270964 5349 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:12:06.185791 7311 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:06.191107 6642 raft_consensus.cc:686] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:06.285835 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:06.286298 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:06.287051 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:06.287206 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:06.324397 5272 raft_consensus.cc:2379] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 2: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:06.287874 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:06.290498 7263 consensus_queue.cc:237] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:06.295403 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 106246us
I20260929 14:12:06.301940 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:06.303068 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:12:06.303972 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 130006us
I20260929 14:12:06.165511 7235 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:06.311890 5270 raft_consensus.cc:2379] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 4: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:06.317081 5164 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:06.327718 5269 raft_consensus.cc:2379] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 5: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:06.328912 5195 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:06.330327 5242 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:06.331025 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 165303us
I20260929 14:12:06.328338 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:06.340816 5268 raft_consensus.cc:2379] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 5: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:06.351763 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:06.323591 7311 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:06.362857 5316 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:06.371080 5197 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
W20260929 14:12:06.376030 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 45789.9us
I20260929 14:12:06.352885 5346 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:06.398252 5242 leader_election.cc:398] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Vote denied by peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497) with higher term. Message: Invalid argument: T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:06.354260 7198 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:06.395130 5346 raft_consensus.cc:595] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:06.357975 7326 raft_consensus.cc:2781] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:12:06.406244 7277 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:06.354003 7235 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:06.410236 7368 log.cc:924] Time spent T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.098s user 0.000s sys 0.000s
I20260929 14:12:06.410799 6642 consensus_queue.cc:237] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:06.421564 7041 consensus_queue.cc:1046] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:06.420946 5316 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:06.425004 7313 raft_consensus.cc:979] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:06.389357 7321 consensus_queue.cc:1035] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.045s
I20260929 14:12:06.434381 5346 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:06.474862 5346 raft_consensus.cc:595] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:06.439833 5314 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:06.467248 5165 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:06.416966 7209 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 8 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:06.446700 7277 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:06.478085 5902 raft_consensus.cc:2379] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 6: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:12:06.478179 7277 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
W20260929 14:12:06.482504 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 61637.5us
W20260929 14:12:06.488555 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 111307us
I20260929 14:12:06.490244 7318 raft_consensus.cc:2781] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:12:06.494585 6917 raft_consensus.cc:2781] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:06.495132 7354 raft_consensus.cc:1067] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:06.513139 5242 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:06.513911 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 235441us
I20260929 14:12:06.524119 5268 raft_consensus.cc:1261] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260929 14:12:06.528879 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:06.532470 5195 raft_consensus.cc:1261] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:12:06.539624 5346 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:06.540349 5346 raft_consensus.cc:595] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:06.540736 5346 raft_consensus.cc:603] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260929 14:12:06.557883 7370 log.cc:924] Time spent T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20260929 14:12:06.562000 7209 consensus_queue.cc:1035] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:06.562898 5349 raft_consensus.cc:1261] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:12:06.508409 7297 consensus_queue.cc:1046] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260929 14:12:06.522078 6917 raft_consensus.cc:686] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:06.521275 7318 raft_consensus.cc:686] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:06.564893 7269 raft_consensus.cc:3032] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:12:06.513870 7326 raft_consensus.cc:686] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:06.522706 7276 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:06.566067 6917 consensus_queue.cc:237] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:06.574090 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:06.576171 5346 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:06.547924 7205 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:06.602867 5346 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 8.
I20260929 14:12:06.549050 7267 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:06.615000 7041 consensus_queue.cc:1035] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:12:06.617600 5195 raft_consensus.cc:1261] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:12:06.618515 5199 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:12:06.627609 5199 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:06.628082 5199 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:06.569751 7269 raft_consensus.cc:729] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:06.639577 5198 raft_consensus.cc:1226] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 1. Current term is 2. Ops: []
I20260929 14:12:06.641611 5346 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:06.642331 5346 raft_consensus.cc:595] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:06.642799 5346 raft_consensus.cc:603] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:12:06.644994 7320 consensus_queue.cc:1035] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:12:06.634184 7205 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:06.647353 7205 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:06.647853 7205 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:06.648178 7235 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:06.648386 7205 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:06.650043 7269 consensus_queue.cc:260] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:06.653327 5346 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:06.655702 7198 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:06.641249 5199 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:06.672597 5346 raft_consensus.cc:595] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:06.692440 5199 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:06.700639 7269 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:06.641428 7267 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:06.703420 7267 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:06.577045 7276 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:06.708001 7276 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:06.709327 7311 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:06.710093 7353 consensus_meta.cc:287] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.562s user 0.003s sys 0.007s
I20260929 14:12:06.710861 7353 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:06.663708 7318 consensus_queue.cc:237] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:06.673463 7277 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:06.693398 6868 consensus_queue.cc:1046] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260929 14:12:06.716751 7276 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:06.716910 5902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:06.717463 7276 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:06.718000 5902 raft_consensus.cc:2379] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 6: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:12:06.718940 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:06.719414 6991 raft_consensus.cc:3032] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 14:12:06.719797 6991 raft_consensus.cc:729] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:06.720605 6991 consensus_queue.cc:260] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:06.721737 6991 raft_consensus.cc:3037] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:12:06.731525 5194 raft_consensus.cc:1261] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260929 14:12:06.716110 7277 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:06.697417 7198 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:06.740473 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 187190us
I20260929 14:12:06.674247 6717 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:06.741737 5271 raft_consensus.cc:1226] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:06.727434 7352 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:06.745550 7381 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260929 14:12:06.746754 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 236879us
I20260929 14:12:06.674054 7205 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:06.719408 7276 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:06.764025 5268 raft_consensus.cc:1261] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260929 14:12:06.765144 5342 raft_consensus.cc:1261] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260929 14:12:06.765174 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:06.766558 5347 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 6.
I20260929 14:12:06.773684 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:06.774298 5194 raft_consensus.cc:3037] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 4
I20260929 14:12:06.742534 6822 consensus_queue.cc:1035] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.024s
I20260929 14:12:06.749707 5269 raft_consensus.cc:1226] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:06.783867 5349 raft_consensus.cc:1261] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260929 14:12:06.748574 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 278845us
W20260929 14:12:06.761458 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 294267us
I20260929 14:12:06.725625 7326 consensus_queue.cc:237] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:06.755957 5242 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:06.817528 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:06.825752 5165 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:06.766697 7235 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:06.746987 7198 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:06.816818 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:06.840689 5342 raft_consensus.cc:3032] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:06.840835 5269 raft_consensus.cc:1261] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:12:06.841043 5342 raft_consensus.cc:729] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:06.830806 5198 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:06.778163 5870 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:12:06.842358 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 64269.8us
I20260929 14:12:06.842984 5870 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:06.827823 5194 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:06.759232 7352 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:06.844523 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:06.855644 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 99755us
I20260929 14:12:06.818060 7353 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:06.851052 5316 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:12:06.923837 5242 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:06.939352 5269 raft_consensus.cc:1261] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:12:06.848104 5342 consensus_queue.cc:260] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:06.830278 7325 consensus_queue.cc:1035] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
W20260929 14:12:06.952313 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 101381us
I20260929 14:12:06.953205 5316 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:06.912576 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:06.920378 7189 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Leader election won for term 8
W20260929 14:12:06.949052 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 106033us
I20260929 14:12:06.967548 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:06.968204 5198 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 5
I20260929 14:12:06.968704 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:06.969324 5192 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:06.925567 7381 raft_consensus.cc:2726] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:12:06.981429 5342 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:06.938619 7312 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:06.846513 7235 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:06.944121 7276 raft_consensus.cc:2726] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:06.949653 7378 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:06.972532 7041 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:06.993659 5268 raft_consensus.cc:1261] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:12:06.949002 7391 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:07.005367 5198 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:06.951705 7321 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:07.005826 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:07.031858 5269 raft_consensus.cc:3037] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:06.993511 7383 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:07.039505 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 226289us
W20260929 14:12:07.041505 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 52391.8us
I20260929 14:12:07.017639 5242 leader_election.cc:398] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497) with higher term. Message: Invalid argument: T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:06.950852 5902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:07.051185 7405 consensus_queue.cc:1046] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:07.052615 7405 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:07.020066 7389 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:12:07.053097 7405 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:07.053779 7389 raft_consensus.cc:3032] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:07.054142 7389 raft_consensus.cc:729] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:07.056449 5902 raft_consensus.cc:2379] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:07.054483 5275 raft_consensus.cc:1261] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260929 14:12:07.005359 7312 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:07.057478 7273 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:07.058264 7273 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:06.999864 7357 raft_consensus.cc:2781] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:12:07.061571 7357 raft_consensus.cc:686] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:07.062234 7041 raft_consensus.cc:686] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:07.062326 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:07.062433 7357 consensus_queue.cc:237] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:07.063220 7041 consensus_queue.cc:237] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:07.034077 7302 consensus_queue.cc:1035] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260929 14:12:07.046384 7391 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:07.067543 7391 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:07.068086 7383 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:07.071048 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:07.071753 5200 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:07.038224 5271 raft_consensus.cc:1261] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:12:07.076679 5272 raft_consensus.cc:1261] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:12:07.078233 5165 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:07.058935 7312 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:07.016000 5192 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:07.090484 5902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:07.011204 7378 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 14:12:07.094305 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 52026.1us
I20260929 14:12:07.020315 7311 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:07.012296 7189 raft_consensus.cc:686] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 8 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
W20260929 14:12:07.115073 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 113624us
I20260929 14:12:07.149739 7335 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:07.151566 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 111084us
I20260929 14:12:07.155165 5165 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:07.155260 5242 leader_election.cc:398] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Vote denied by peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497) with higher term. Message: Invalid argument: T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:07.155755 5242 leader_election.cc:401] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260929 14:12:07.156767 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:07.157660 5907 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:07.159266 5167 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:12:07.170814 5242 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:07.179792 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:07.180600 5345 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
W20260929 14:12:07.183010 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 115162us
I20260929 14:12:07.047050 6717 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:07.185571 5342 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:07.189965 5164 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:07.190829 5165 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:07.194214 5316 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:07.194679 5269 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
W20260929 14:12:07.196549 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 229439us
I20260929 14:12:07.152657 7205 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:07.162036 7378 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:07.160498 6642 raft_consensus.cc:2726] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:07.194914 7152 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.503s user 0.005s sys 0.017s
I20260929 14:12:07.178177 7267 raft_consensus.cc:2781] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:07.221804 6642 raft_consensus.cc:2726] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:07.222004 7267 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:07.222604 7267 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:07.222745 7325 consensus_queue.cc:1035] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:07.217752 7040 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:07.225136 7040 raft_consensus.cc:686] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:07.226013 7040 consensus_queue.cc:237] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:07.178383 7263 consensus_queue.cc:1035] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.051s
W20260929 14:12:07.231237 7269 consensus_meta.cc:287] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.530s user 0.010s sys 0.000s
I20260929 14:12:07.231766 7205 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:07.232326 7205 raft_consensus.cc:686] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:07.233157 7205 consensus_queue.cc:237] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:07.011988 7381 consensus_queue.cc:1035] T 88e8219d06734c04acdce83c7b9c1c53 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.181s
I20260929 14:12:07.166960 7276 raft_consensus.cc:2726] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:07.210384 5316 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:07.046077 7321 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:07.187100 6917 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:07.250324 7321 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:07.250885 7321 consensus_queue.cc:1035] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:12:07.205099 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:07.182595 7335 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:07.269904 5873 leader_election.cc:334] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.661s (SENT)
I20260929 14:12:07.270399 5873 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:07.267166 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:07.239360 7273 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:07.238412 7352 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:07.274205 5274 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
W20260929 14:12:07.274479 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 64217.7us
I20260929 14:12:07.278215 5871 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
W20260929 14:12:07.276158 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 85438.9us
W20260929 14:12:07.283133 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 93256.1us
I20260929 14:12:07.253360 7404 raft_consensus.cc:2726] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497) with higher term. Message: Invalid argument: T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:07.268539 7209 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:07.266938 7256 consensus_queue.cc:1035] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.012s
I20260929 14:12:07.252305 7311 consensus_queue.cc:1046] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260929 14:12:07.259155 6917 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:07.207628 7389 consensus_queue.cc:260] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:07.353586 5165 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:07.353129 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 71816.6us
I20260929 14:12:07.527614 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:07.432072 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 162170us
I20260929 14:12:07.529044 5316 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:07.650261 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:07.651335 5346 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:12:07.273952 7379 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:07.409552 7407 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:07.403668 7235 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:07.721422 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 398093us
I20260929 14:12:07.658540 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260929 14:12:07.495273 6991 consensus_meta.cc:287] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.773s user 0.011s sys 0.000s
W20260929 14:12:07.795634 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 71578.6us
W20260929 14:12:07.796182 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 267234us
I20260929 14:12:07.753995 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260929 14:12:07.806260 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 452778us
W20260929 14:12:07.809279 5873 leader_election.cc:334] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.615s (SENT)
I20260929 14:12:07.809267 5347 raft_consensus.cc:2454] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:12:07.415155 7381 raft_consensus.cc:2781] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:12:07.278959 7002 consensus_queue.cc:1035] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.463s
W20260929 14:12:07.739930 7393 log.cc:924] Time spent T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.500s user 0.000s sys 0.001s
I20260929 14:12:07.424918 7321 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260929 14:12:07.526304 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 274407us
I20260929 14:12:07.508211 7406 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:07.721469 7407 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 14:12:07.845332 5873 leader_election.cc:334] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.978s (SENT)
I20260929 14:12:07.845829 5873 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:07.454658 6717 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:12:07.835352 7403 log.cc:924] Time spent T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.466s user 0.001s sys 0.000s
I20260929 14:12:07.515625 7408 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:07.722653 7391 raft_consensus.cc:2726] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:07.854794 5873 leader_election.cc:334] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.066s (SENT)
I20260929 14:12:07.883242 5873 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:07.740211 6642 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:07.843786 7205 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.608s user 0.007s sys 0.016s
W20260929 14:12:07.904189 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 60717.3us
I20260929 14:12:07.438374 7352 raft_consensus.cc:465] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Not starting pre-election -- already a leader
I20260929 14:12:07.771015 7404 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:07.835574 7235 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:07.903249 5242 leader_election.cc:334] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.195s (SENT)
I20260929 14:12:07.917388 5242 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:07.866510 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 67358.3us
I20260929 14:12:07.315133 7400 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:07.315083 5902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:07.912631 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 57882.7us
W20260929 14:12:07.914893 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 107624us
I20260929 14:12:07.919850 5165 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:07.926260 5165 leader_election.cc:398] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5.
I20260929 14:12:07.927145 5165 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:07.651934 7385 consensus_queue.cc:1035] T 678eb0db533d4bea872808b6fb39b764 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.253s
I20260929 14:12:07.929427 7352 raft_consensus.cc:2726] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:07.930322 7352 raft_consensus.cc:2726] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:12:07.846984 7321 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:07.916606 6642 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:07.944439 6642 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:07.863474 7313 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:07.912110 7381 raft_consensus.cc:686] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:07.954368 7381 consensus_queue.cc:237] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:07.957640 7335 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:07.958400 7335 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:07.958999 7335 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:07.960587 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:07.962118 6917 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:07.962833 6917 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:07.963382 6917 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:07.965428 7396 log.cc:924] Time spent T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.688s user 0.000s sys 0.001s
I20260929 14:12:07.965627 7388 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:07.966125 7388 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:07.968152 7388 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:07.852392 7418 raft_consensus.cc:3032] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:07.995638 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:12:07.914647 7205 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.679s user 0.008s sys 0.016s
I20260929 14:12:07.940146 7412 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:07.940523 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 57699.2us
I20260929 14:12:07.852478 7407 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:07.863338 7406 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:07.852732 7357 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:07.282291 7189 consensus_queue.cc:237] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:07.864327 7389 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
W20260929 14:12:07.913710 7399 log.cc:924] Time spent T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.601s user 0.000s sys 0.001s
I20260929 14:12:07.996516 5275 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:07.930960 7404 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:07.973403 7417 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:08.015511 7409 log.cc:924] Time spent T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.601s user 0.001s sys 0.000s
I20260929 14:12:07.885556 6717 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:08.017351 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:08.019304 5873 leader_election.cc:334] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.196s (SENT)
I20260929 14:12:08.025635 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:08.079639 5272 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
W20260929 14:12:08.079690 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 60425us
I20260929 14:12:07.937147 7152 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:07.683070 7209 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:07.974284 7321 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:08.046257 7313 raft_consensus.cc:686] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
W20260929 14:12:08.046365 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 127550us
I20260929 14:12:08.107147 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:08.048815 7404 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:07.941004 7235 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:07.996235 7418 raft_consensus.cc:729] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:07.999456 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:08.138803 5268 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:08.005563 7376 raft_consensus.cc:2726] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:08.075872 5275 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:07.796027 7379 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:08.015077 6868 consensus_queue.cc:1035] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.107s
I20260929 14:12:08.132902 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1545) took 1414 ms. Trace:
I20260929 14:12:08.067540 7205 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:08.062832 7335 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:08.159809 5199 rpcz_store.cc:276] 0929 14:12:06.718044 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:06.729203 (+ 11159us) service_pool.cc:227] Handling call
0929 14:12:06.729630 (+ 427us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:06.942012 (+212382us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:06.942040 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:06.942434 (+ 394us) log.cc:842] Serialized 26 byte log entry
0929 14:12:07.179339 (+236905us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:07.179404 (+ 65us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:08.132388 (+952984us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b5400408b54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:08.132451 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:08.132515 (+ 64us) raft_consensus.cc:1671] finished
0929 14:12:08.132534 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:08.132824 (+ 290us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18269,"raft.queue_time_us":19111,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":21658,"spinlock_wait_cycles":445513088,"thread_start_us":236548,"threads_started":1,"wal-append.queue_time_us":321458}
I20260929 14:12:08.037056 7385 consensus_queue.cc:1035] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.101s
I20260929 14:12:08.199435 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1510) took 2141 ms. Trace:
I20260929 14:12:08.181070 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:08.199925 5193 rpcz_store.cc:276] 0929 14:12:06.058162 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:06.068653 (+ 10491us) service_pool.cc:227] Handling call
0929 14:12:06.144476 (+ 75823us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:06.182766 (+ 38290us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:06.184608 (+ 1842us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:06.294644 (+110036us) log.cc:842] Serialized 26 byte log entry
0929 14:12:06.504460 (+209816us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:06.510155 (+ 5695us) log.cc:842] Serialized 14 byte log entry
0929 14:12:06.531406 (+ 21251us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:06.540959 (+ 9553us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:06.654206 (+113247us) spinlock_profiling.cc:243] Waited 3.1 ms on lock 0x7b1800350740. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b45764 00007fbda1b48b11 00007fbda1b48ac9 00007fbda1b48a59 00007fbda1b48a21 00007fbda1b47d1d 00007fbdaada9204 00007fbdaada9039 00007fbda1b27dc6
0929 14:12:08.133741 (+1479535us) spinlock_profiling.cc:243] Waited 4.75 ms on lock 0x7b1800350740. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b45638 00007fbda1b48b11 00007fbda1b48ac9 00007fbda1b48a59 00007fbda1b48a21 00007fbda1b47d1d 00007fbdaada9204 00007fbdaada9039 00007fbda1b27dc6
0929 14:12:08.196657 (+ 62916us) raft_consensus.cc:1671] finished
0929 14:12:08.199183 (+ 2526us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:08.199341 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28100,"prepare.queue_time_us":403,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":54716,"raft.run_cpu_time_us":19780,"raft.run_wall_time_us":102643,"spinlock_wait_cycles":18057856,"thread_start_us":206164,"threads_started":1,"wal-append.queue_time_us":347313}
I20260929 14:12:07.915299 7408 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:08.206429 7205 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:08.215200 7205 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:08.028147 7412 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:08.218219 7412 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:08.218053 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:08.179774 7404 raft_consensus.cc:2726] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:08.151916 7411 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:08.220750 7411 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:08.035302 7357 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:08.227435 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:08.228278 7378 consensus_meta.cc:287] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.998s user 0.012s sys 0.000s
I20260929 14:12:08.228636 5905 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45232 (request call id 385) took 1397 ms. Trace:
I20260929 14:12:08.202670 7313 consensus_queue.cc:237] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:08.078044 7417 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:08.235602 7394 log.cc:924] Time spent T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 1.239s user 0.010s sys 0.019s
I20260929 14:12:08.248625 5201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1543) took 1588 ms. Trace:
I20260929 14:12:08.250005 5201 rpcz_store.cc:276] 0929 14:12:06.660352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:06.660846 (+ 494us) service_pool.cc:227] Handling call
0929 14:12:06.675689 (+ 14843us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:06.676063 (+ 374us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:06.676104 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:06.676603 (+ 499us) log.cc:842] Serialized 26 byte log entry
0929 14:12:06.796018 (+119415us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:06.796087 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:06.796184 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:08.234776 (+1438592us) spinlock_profiling.cc:243] Waited 8.05 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda46efbee 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551 00007fbda46f8509 00007fbda46f8499
0929 14:12:08.248335 (+ 13559us) raft_consensus.cc:1671] finished
0929 14:12:08.248356 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:08.248557 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":14336,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":13520,"spinlock_wait_cycles":18513792,"thread_start_us":94566,"threads_started":1,"wal-append.queue_time_us":214233}
I20260929 14:12:08.252996 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:08.263303 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:08.120776 7428 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:08.181593 7267 consensus_meta.cc:287] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.958s user 0.009s sys 0.004s
I20260929 14:12:08.289821 5269 raft_consensus.cc:1261] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:12:08.295138 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:08.248035 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:08.300594 5907 raft_consensus.cc:2454] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:08.301050 5349 raft_consensus.cc:1261] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:12:08.307271 7412 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:08.300052 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:08.298494 5196 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.706s user 0.010s sys 0.020s
I20260929 14:12:08.307674 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:08.304289 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:08.329246 5197 raft_consensus.cc:2379] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 4: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:08.307456 7379 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:08.307835 7428 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:08.232583 7404 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:08.328550 5196 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.804s user 0.010s sys 0.020s
I20260929 14:12:08.206763 6717 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:08.229352 7408 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:08.248425 7235 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:08.337073 5201 raft_consensus.cc:1261] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:12:08.228734 7378 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:08.231791 7406 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:08.354980 5345 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.060s sys 0.078s
W20260929 14:12:08.356182 5345 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.511s user 0.060s sys 0.078s
I20260929 14:12:08.332005 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:08.354801 7326 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.890s user 0.002s sys 0.025s
I20260929 14:12:08.406236 5268 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
W20260929 14:12:08.419732 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 283025us
I20260929 14:12:08.251772 5905 rpcz_store.cc:276] 0929 14:12:06.830804 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:06.839688 (+ 8884us) service_pool.cc:227] Handling call
0929 14:12:08.227220 (+1387532us) spinlock_profiling.cc:243] Waited 3.2 ms on lock 0x7b5800027698. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:12:08.228534 (+ 1314us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7357696}
I20260929 14:12:08.249593 7357 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:08.354077 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:08.324944 7267 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:08.376310 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:08.410373 7326 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.942s user 0.002s sys 0.026s
I20260929 14:12:08.403127 7406 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:08.427748 5343 raft_consensus.cc:2396] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260929 14:12:08.345397 7412 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:08.427443 5902 raft_consensus.cc:1226] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 6. Current term is 7. Ops: []
W20260929 14:12:08.344352 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 417247us
I20260929 14:12:08.470810 5198 raft_consensus.cc:2354] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6.
I20260929 14:12:08.371213 7404 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:08.472306 5167 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:08.476723 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:08.480468 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:08.486423 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:08.210517 7418 consensus_queue.cc:260] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:08.487231 5267 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:12:08.508589 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:08.509264 5267 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:12:08.510167 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:08.510776 5267 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:08.519083 7198 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:08.524667 7387 log.cc:924] Time spent T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 1.602s user 0.004s sys 0.016s
W20260929 14:12:08.524403 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 219296us
I20260929 14:12:08.535043 7308 raft_consensus.cc:979] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:08.431492 7411 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:08.507068 5165 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:08.540853 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 68644.4us
W20260929 14:12:08.541265 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 69064.3us
I20260929 14:12:08.545047 5871 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:08.442806 5193 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:08.554177 5900 raft_consensus.cc:2379] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 5: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
W20260929 14:12:08.583273 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 112013us
I20260929 14:12:08.567055 7198 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:08.568674 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:08.589427 5904 raft_consensus.cc:2396] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260929 14:12:08.588928 7198 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:08.591045 7389 consensus_meta.cc:287] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.551s user 0.009s sys 0.000s
I20260929 14:12:08.591748 7198 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:08.592514 7432 raft_consensus.cc:2726] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:08.540225 7422 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:08.459996 7321 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:08.596997 5316 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:08.592440 5167 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:08.462286 7406 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:08.593845 7414 log.cc:924] Time spent T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.181s user 0.000s sys 0.000s
I20260929 14:12:08.582024 7209 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:08.582813 7263 raft_consensus.cc:2781] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:08.572762 7431 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:08.573380 7418 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:08.565167 7411 raft_consensus.cc:2726] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:08.555106 7326 consensus_queue.cc:1035] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260929 14:12:08.572975 7205 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:08.619634 7405 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:08.666097 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 69180.7us
I20260929 14:12:08.617991 7422 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:08.650671 7419 log.cc:924] Time spent T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.380s user 0.000s sys 0.001s
W20260929 14:12:09.014359 5195 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.034s sys 0.056s
W20260929 14:12:08.854586 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 395268us
I20260929 14:12:08.600939 7320 raft_consensus.cc:1067] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Signalling peer c5ec116e39144881bdf1f2cd3e923cf5 to start an election
W20260929 14:12:08.651273 7425 log.cc:924] Time spent T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.213s user 0.000s sys 0.001s
W20260929 14:12:09.010955 5314 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.515s user 0.007s sys 0.017s
W20260929 14:12:09.170560 5314 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.675s user 0.010s sys 0.019s
W20260929 14:12:08.879601 5199 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.011s sys 0.049s
W20260929 14:12:09.171501 5199 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.846s user 0.017s sys 0.065s
W20260929 14:12:09.161665 5195 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s user 0.035s sys 0.080s
I20260929 14:12:09.148660 5276 raft_consensus.cc:1261] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260929 14:12:08.945320 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 45239.8us
I20260929 14:12:08.653908 7428 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:09.163147 5268 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.683s user 0.003s sys 0.012s
W20260929 14:12:09.176441 5268 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.697s user 0.003s sys 0.012s
W20260929 14:12:08.950428 7407 consensus_meta.cc:287] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.910s user 0.006s sys 0.005s
W20260929 14:12:09.076282 5346 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.827s user 0.005s sys 0.008s
W20260929 14:12:09.180729 5346 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.931s user 0.005s sys 0.012s
I20260929 14:12:08.617597 7276 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:08.659013 7413 log.cc:924] Time spent T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.274s user 0.000s sys 0.000s
I20260929 14:12:08.647275 7415 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:09.173851 5242 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.670s (SENT) [suppressed 49 similar messages]
W20260929 14:12:09.191710 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 46448.1us
I20260929 14:12:09.172544 5195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1568) took 1068 ms. Trace:
W20260929 14:12:08.950668 7434 log.cc:924] Time spent T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260929 14:12:09.142203 5165 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:08.654417 7445 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:08.662339 7438 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:08.657829 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:08.654873 7267 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:08.663939 7395 log.cc:924] Time spent T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 1.636s user 0.016s sys 0.053s
W20260929 14:12:09.163232 5898 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.546s user 0.000s sys 0.009s
I20260929 14:12:09.171635 7411 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:08.879915 7440 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:08.663512 7426 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:08.663559 7427 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:08.946036 6642 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:08.651644 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:08.755963 7417 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:08.677054 7263 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:09.250929 7263 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:08.856405 7378 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:09.252220 5346 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 207) took 1554 ms. Trace:
I20260929 14:12:09.252566 5346 rpcz_store.cc:276] 0929 14:12:07.697484 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:07.697681 (+ 197us) service_pool.cc:227] Handling call
0929 14:12:07.734700 (+ 37019us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:08.152924 (+418224us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:08.152968 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:08.249157 (+ 96189us) log.cc:842] Serialized 26 byte log entry
0929 14:12:09.181315 (+932158us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:09.181352 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:09.181426 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:09.252011 (+ 70585us) raft_consensus.cc:1671] finished
0929 14:12:09.252042 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:09.252167 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":155680,"raft.queue_time_us":128578,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":30022,"thread_start_us":1239688,"threads_started":2,"wal-append.queue_time_us":951031}
I20260929 14:12:08.690069 7406 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:08.945990 7404 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:08.871481 7321 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:09.013118 7277 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:09.220255 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:09.192272 5195 rpcz_store.cc:276] 0929 14:12:08.104350 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:08.108498 (+ 4148us) service_pool.cc:227] Handling call
0929 14:12:08.108827 (+ 329us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:08.109038 (+ 211us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:12:08.462711 (+353673us) log.cc:842] Serialized 14 byte log entry
0929 14:12:09.172182 (+709471us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:09.172237 (+ 55us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:09.172294 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:09.172359 (+ 65us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:09.172490 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":465902,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":25700,"thread_start_us":1036379,"threads_started":2}
I20260929 14:12:09.198189 5276 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 8
I20260929 14:12:08.945990 7430 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:09.220161 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 47352.2us
W20260929 14:12:09.223428 7341 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.017s sys 0.029s
W20260929 14:12:09.201185 7189 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.537s user 0.003s sys 0.014s
W20260929 14:12:09.186376 7381 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.704s user 0.009s sys 0.024s
W20260929 14:12:09.211280 5898 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.613s user 0.000s sys 0.010s
I20260929 14:12:09.235843 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.238692 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:09.239236 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 46154.6us
I20260929 14:12:09.209384 5343 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 6
W20260929 14:12:09.308387 5873 leader_election.cc:334] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.526s (SENT)
I20260929 14:12:09.311261 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1552) took 2346 ms. Trace:
I20260929 14:12:09.257802 5904 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
W20260929 14:12:09.313975 7458 log.cc:924] Time spent T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260929 14:12:09.162976 7205 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:09.220659 7267 raft_consensus.cc:2726] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:09.255466 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.256424 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:08.946877 7412 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:09.260012 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.226701 7276 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:09.226418 7411 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:09.279548 5342 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 203) took 2065 ms. Trace:
W20260929 14:12:09.260531 7041 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.169s user 0.030s sys 0.060s
W20260929 14:12:09.341557 7041 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.250s user 0.031s sys 0.061s
I20260929 14:12:09.339390 5342 rpcz_store.cc:276] 0929 14:12:07.214484 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:07.214628 (+ 144us) service_pool.cc:227] Handling call
0929 14:12:07.215028 (+ 400us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:07.215443 (+ 415us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:07.215480 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:07.260205 (+ 44725us) log.cc:842] Serialized 26 byte log entry
0929 14:12:07.722653 (+462448us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:07.722704 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:07.722786 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:09.279334 (+1556548us) raft_consensus.cc:1671] finished
0929 14:12:09.279356 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:09.279488 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65,"raft.queue_time_us":109,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":3600,"thread_start_us":370331,"threads_started":1,"wal-append.queue_time_us":1068526}
I20260929 14:12:09.344321 5343 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:09.249096 7428 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:09.320348 5873 leader_election.cc:334] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.633s (SENT)
I20260929 14:12:09.347947 7411 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:09.348435 5873 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.249931 7426 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:09.359432 7426 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:09.311695 5196 rpcz_store.cc:276] 0929 14:12:06.964776 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:06.964940 (+ 164us) service_pool.cc:227] Handling call
0929 14:12:06.995705 (+ 30765us) raft_consensus.cc:1457] Updating replica for 2 ops
0929 14:12:06.996095 (+ 390us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:06.996126 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0929 14:12:07.001926 (+ 5800us) log.cc:842] Serialized 52 byte log entry
0929 14:12:07.352713 (+350787us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:08.338298 (+985585us) log.cc:842] Serialized 14 byte log entry
0929 14:12:08.338543 (+ 245us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:08.338629 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:09.311009 (+972380us) raft_consensus.cc:1671] finished
0929 14:12:09.311041 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:09.311198 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1197499,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":181,"raft.run_cpu_time_us":351,"raft.run_wall_time_us":30162,"thread_start_us":1163960,"threads_started":2,"wal-append.queue_time_us":1287163}
W20260929 14:12:09.280844 7381 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.818s user 0.009s sys 0.025s
W20260929 14:12:09.280923 7341 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.628s user 0.017s sys 0.031s
I20260929 14:12:09.163444 7431 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:09.183342 7149 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:09.336458 5240 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.588s user 0.016s sys 0.014s
W20260929 14:12:09.383287 5240 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.831s user 0.017s sys 0.015s
I20260929 14:12:09.335211 5316 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:09.325069 5351 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
W20260929 14:12:09.385674 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 50552.4us
I20260929 14:12:09.338728 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.182834 7407 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:09.200613 7415 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:09.404615 5196 raft_consensus.cc:1261] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:12:09.405871 5192 raft_consensus.cc:1261] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:12:09.406318 7415 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:09.406944 7415 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:09.407529 7415 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:09.408694 5197 raft_consensus.cc:1261] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:12:09.205243 7445 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:09.409646 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:09.410013 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:09.410645 5904 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Advancing to term 8
I20260929 14:12:09.374734 7428 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:09.410912 7445 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:09.411692 5345 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 204) took 1886 ms. Trace:
I20260929 14:12:09.246789 7427 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:09.307585 5314 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
W20260929 14:12:09.278827 7189 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s user 0.004s sys 0.014s
I20260929 14:12:09.257551 6642 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:09.328164 7326 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:09.270143 7277 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:09.356566 7422 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:09.355291 5164 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:09.432214 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 77051.9us
I20260929 14:12:09.236647 7440 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:09.353869 5902 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45248 (request call id 321) took 1252 ms. Trace:
I20260929 14:12:09.353659 7276 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:09.351483 5898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45252 (request call id 452) took 1627 ms. Trace:
I20260929 14:12:09.355580 6917 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:09.281682 5900 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:09.373618 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.338326 7452 consensus_queue.cc:1035] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:12:09.214960 7438 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:09.441437 5902 rpcz_store.cc:276] 0929 14:12:08.101316 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:08.152238 (+ 50922us) service_pool.cc:227] Handling call
0929 14:12:08.198999 (+ 46761us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:08.506503 (+307504us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:09.353613 (+847110us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b54000cbcd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:09.353803 (+ 190us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":455465216}
I20260929 14:12:09.390205 5239 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:09.401196 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.402128 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.402403 7431 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:09.402807 7149 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:09.403690 7041 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:09.404187 7407 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:09.412041 5345 rpcz_store.cc:276] 0929 14:12:07.525323 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:07.525438 (+ 115us) service_pool.cc:227] Handling call
0929 14:12:07.752700 (+227262us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:08.370928 (+618228us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:08.370971 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:08.522045 (+151074us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b2c000339d8. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda3450e11 00007fbda3460442 00007fbda4aa8c11 00007fbda4a6d58a 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772
0929 14:12:08.522531 (+ 486us) log.cc:842] Serialized 26 byte log entry
0929 14:12:09.011572 (+489041us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:09.011624 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:09.011734 (+ 110us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:09.411415 (+399681us) raft_consensus.cc:1671] finished
0929 14:12:09.411435 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:09.411636 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":167329,"raft.queue_time_us":130249,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":26223,"spinlock_wait_cycles":3618048,"thread_start_us":1014400,"threads_started":2,"wal-append.queue_time_us":852315}
W20260929 14:12:09.421896 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 114441us
I20260929 14:12:09.484187 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:09.441978 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:09.484905 5906 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 7.
I20260929 14:12:09.430262 7460 raft_consensus.cc:2781] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:12:09.489204 5242 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:09.463312 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.471349 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:09.490338 5346 raft_consensus.cc:2454] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:12:09.490577 5200 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
W20260929 14:12:09.478765 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 88639.3us
I20260929 14:12:09.466164 5902 raft_consensus.cc:1261] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 14:12:09.492044 7273 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:09.495653 7445 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:09.496099 7445 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:09.495934 7273 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:09.466660 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:09.496472 7445 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:09.495378 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:09.462179 7205 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:09.497756 7263 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:09.498572 7477 log.cc:924] Time spent T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.151s user 0.000s sys 0.001s
I20260929 14:12:09.499116 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:09.500219 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.500451 7263 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:09.501488 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 359347us
I20260929 14:12:09.464381 7438 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:09.465121 7440 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:09.502348 5165 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.503175 7205 raft_consensus.cc:2726] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:09.503620 7440 consensus_queue.cc:1035] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:09.504379 5165 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.504753 5904 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:09.505199 7205 raft_consensus.cc:2726] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:09.507305 5165 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.498365 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:09.512655 5242 leader_election.cc:334] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.503s (SENT)
I20260929 14:12:09.514925 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.515722 5344 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:12:09.492202 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1576) took 1217 ms. Trace:
I20260929 14:12:09.517722 5239 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:12:09.517254 5199 rpcz_store.cc:276] 0929 14:12:08.274534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:08.274697 (+ 163us) service_pool.cc:227] Handling call
0929 14:12:08.275097 (+ 400us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:08.275348 (+ 251us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:12:09.172128 (+896780us) log.cc:842] Serialized 14 byte log entry
0929 14:12:09.491509 (+319381us) log.cc:842] Serialized 14 byte log entry
0929 14:12:09.491776 (+ 267us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:09.491840 (+ 64us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:09.491903 (+ 63us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:09.492002 (+ 99us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:09.492133 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":387657,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":5339,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":24991,"spinlock_wait_cycles":90152064,"thread_start_us":1154816,"threads_started":2,"wal-append.queue_time_us":323854}
I20260929 14:12:09.519411 7440 raft_consensus.cc:2781] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:09.518029 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.519944 7440 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:09.513824 5242 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.520617 7440 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:09.484328 7041 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:09.521643 7041 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:09.521420 7438 raft_consensus.cc:2726] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:12:09.523680 5900 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:09.475589 7415 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:09.528880 7422 raft_consensus.cc:2726] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:12:09.485247 6917 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:09.531232 7205 raft_consensus.cc:2726] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:09.530649 7445 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:09.531193 6917 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:09.534467 7427 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:09.536101 6642 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:09.537081 5907 raft_consensus.cc:2379] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 4: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:12:09.539412 5268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 249) took 1060 ms. Trace:
I20260929 14:12:09.539822 5268 rpcz_store.cc:276] 0929 14:12:08.478535 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:08.478987 (+ 452us) service_pool.cc:227] Handling call
0929 14:12:08.479316 (+ 329us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:09.176870 (+697554us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:09.176898 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:09.182964 (+ 6066us) log.cc:842] Serialized 26 byte log entry
0929 14:12:09.279359 (+ 96395us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:09.279414 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:09.279527 (+ 113us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:09.539178 (+259651us) raft_consensus.cc:1671] finished
0929 14:12:09.539199 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:09.539349 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":170735,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":51713,"thread_start_us":752033,"threads_started":2,"wal-append.queue_time_us":153069}
I20260929 14:12:09.546195 5316 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:09.552021 5350 raft_consensus.cc:2354] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 7. Current term is 8.
I20260929 14:12:09.549657 5200 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
W20260929 14:12:09.591585 5242 leader_election.cc:334] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.542s (SENT)
I20260929 14:12:09.592214 5242 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:09.593596 5242 leader_election.cc:334] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.553s (SENT)
W20260929 14:12:09.594605 5242 leader_election.cc:334] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.574s (SENT)
I20260929 14:12:09.441643 7469 raft_consensus.cc:2781] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:09.470710 7276 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:09.470347 7428 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:09.431452 7326 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:09.656564 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 110484us
W20260929 14:12:09.659701 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 174902us
I20260929 14:12:09.670868 5314 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:09.488402 7464 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260929 14:12:09.662034 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 120245us
W20260929 14:12:09.661644 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 67057.2us
I20260929 14:12:09.661635 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.529368 7440 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:09.674232 5242 leader_election.cc:334] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.575s (SENT)
I20260929 14:12:09.657049 7427 raft_consensus.cc:2726] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260929 14:12:09.665709 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 311523us
W20260929 14:12:09.672315 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 165083us
I20260929 14:12:09.680294 5165 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:09.545260 7460 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:09.633987 7411 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:09.681255 5275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1578) took 1187 ms. Trace:
W20260929 14:12:09.681875 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 361531us
W20260929 14:12:09.682691 5242 leader_election.cc:334] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.588s (SENT)
I20260929 14:12:09.662354 6642 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:09.685058 5165 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.661371 6917 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:09.686622 5242 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:09.689971 5242 leader_election.cc:334] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.612s (SENT)
I20260929 14:12:09.690864 5242 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.690989 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:09.691936 5242 leader_election.cc:334] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.648s (SENT)
I20260929 14:12:09.692302 5903 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45252 (request call id 455) took 1144 ms. Trace:
I20260929 14:12:09.686249 5275 rpcz_store.cc:276] 0929 14:12:08.493526 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:08.493684 (+ 158us) service_pool.cc:227] Handling call
0929 14:12:08.494093 (+ 409us) raft_consensus.cc:1457] Updating replica for 2 ops
0929 14:12:08.603636 (+109543us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:08.603685 (+ 49us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0929 14:12:08.642225 (+ 38540us) log.cc:842] Serialized 52 byte log entry
0929 14:12:09.302069 (+659844us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:09.679291 (+377222us) log.cc:842] Serialized 14 byte log entry
0929 14:12:09.679665 (+ 374us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:09.679742 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:09.679786 (+ 44us) raft_consensus.cc:1671] finished
0929 14:12:09.679797 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:09.681175 (+ 1378us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":161834,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":159704,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"thread_start_us":558622,"threads_started":3,"wal-append.queue_time_us":357501}
I20260929 14:12:09.692559 5903 rpcz_store.cc:276] 0929 14:12:08.548012 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:08.548261 (+ 249us) service_pool.cc:227] Handling call
0929 14:12:09.691624 (+1143363us) spinlock_profiling.cc:243] Waited 2.14 ms on lock 0x7b5800027698. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:12:09.691798 (+ 174us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:09.692007 (+ 209us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:09.692033 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:09.692074 (+ 41us) raft_consensus.cc:1631] Marking committed up to 6
0929 14:12:09.692105 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:09.692155 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:09.692248 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4925184}
I20260929 14:12:09.675160 7464 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:09.668267 7320 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:09.675016 7428 raft_consensus.cc:2781] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:09.668354 7469 raft_consensus.cc:686] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:09.698475 5316 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:09.699177 5314 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:09.699309 5242 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.700309 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:09.687664 7438 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:09.755429 5903 raft_consensus.cc:2379] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 7: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
W20260929 14:12:09.759855 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 74876.4us
I20260929 14:12:09.760957 5165 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.696247 7205 raft_consensus.cc:2726] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260929 14:12:09.763110 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 71204us
W20260929 14:12:09.763892 5242 leader_election.cc:334] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.654s (SENT)
I20260929 14:12:09.764344 5242 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.690536 7411 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:09.702193 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:09.696189 7459 raft_consensus.cc:2726] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:09.685889 7460 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:12:09.698845 5347 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47252 (request call id 1872) took 2631 ms (client timeout 3286 ms). Trace:
I20260929 14:12:09.749814 7452 raft_consensus.cc:2781] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:09.693306 7427 raft_consensus.cc:2726] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260929 14:12:09.705103 7468 log.cc:924] Time spent T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.343s user 0.000s sys 0.001s
I20260929 14:12:09.441298 5898 rpcz_store.cc:276] 0929 14:12:07.723681 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:07.740167 (+ 16486us) service_pool.cc:227] Handling call
0929 14:12:07.863187 (+123020us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:08.011368 (+148181us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:08.027902 (+ 16534us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:08.133424 (+105522us) log.cc:842] Serialized 26 byte log entry
0929 14:12:08.470786 (+337362us) raft_consensus.cc:1631] Marking committed up to 7
0929 14:12:09.333252 (+862466us) log.cc:842] Serialized 14 byte log entry
0929 14:12:09.351136 (+ 17884us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:09.351228 (+ 92us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:09.351280 (+ 52us) raft_consensus.cc:1671] finished
0929 14:12:09.351298 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:09.351423 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":801644,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":204242,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":108560,"thread_start_us":1009389,"threads_started":2,"wal-append.queue_time_us":463899}
I20260929 14:12:09.702296 7464 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:09.775406 7406 raft_consensus.cc:2726] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:12:09.767162 5167 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.781955 5274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1573) took 1778 ms. Trace:
I20260929 14:12:09.769505 5165 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.762595 7415 raft_consensus.cc:2726] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:09.774418 5242 leader_election.cc:334] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.718s (SENT)
W20260929 14:12:09.786535 5242 leader_election.cc:334] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.720s (SENT)
I20260929 14:12:09.787410 5242 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.706725 7428 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:12:09.775722 5347 rpcz_store.cc:269] 0929 14:12:07.061717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:07.061933 (+ 216us) service_pool.cc:227] Handling call
0929 14:12:09.692437 (+2630504us) spinlock_profiling.cc:243] Waited 746 ms on lock 0x7b540027f954. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:09.692523 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1714962176}
I20260929 14:12:09.751152 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:09.754971 7475 log.cc:924] Time spent T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260929 14:12:09.695719 6642 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:09.703349 7276 raft_consensus.cc:2726] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:09.782874 5167 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.702102 5870 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:09.832218 7318 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:09.706358 7320 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:09.703776 7308 consensus_queue.cc:1046] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.074s
W20260929 14:12:09.791345 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 92947.1us
I20260929 14:12:09.843340 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:09.844131 5275 raft_consensus.cc:2379] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 4: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
W20260929 14:12:09.792143 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 93041.8us
W20260929 14:12:09.847285 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 145239us
I20260929 14:12:09.844376 5164 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:12:09.768684 7267 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:09.795837 7428 raft_consensus.cc:3037] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:09.810724 7427 raft_consensus.cc:2726] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260929 14:12:09.811332 5242 leader_election.cc:334] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.737s (SENT)
I20260929 14:12:09.832113 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:09.782330 5274 rpcz_store.cc:276] 0929 14:12:08.003848 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:08.004150 (+ 302us) service_pool.cc:227] Handling call
0929 14:12:08.013998 (+ 9848us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:08.265853 (+251855us) raft_consensus.cc:1520] Early marking committed up to index 2
0929 14:12:08.872095 (+606242us) log.cc:842] Serialized 14 byte log entry
0929 14:12:09.416463 (+544368us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:09.429629 (+ 13166us) log.cc:842] Serialized 26 byte log entry
0929 14:12:09.429774 (+ 145us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:09.429820 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:09.429908 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:09.781752 (+351844us) raft_consensus.cc:1671] finished
0929 14:12:09.781770 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:09.781901 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1537,"prepare.queue_time_us":1161820,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":152514,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":5013,"thread_start_us":1096609,"threads_started":3,"wal-append.queue_time_us":783701}
W20260929 14:12:09.833640 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 50859.7us
I20260929 14:12:09.777649 7205 raft_consensus.cc:2726] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:09.779145 7452 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:09.854161 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:09.770464 7438 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:09.813776 7431 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:09.832858 7318 raft_consensus.cc:686] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:09.842161 7432 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:09.846900 7476 raft_consensus.cc:2726] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:09.855988 7445 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:09.872170 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 60873.1us
W20260929 14:12:09.875463 5242 leader_election.cc:334] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.737s (SENT)
I20260929 14:12:09.875846 5242 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:09.859411 7209 raft_consensus.cc:2781] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:09.862305 7452 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
W20260929 14:12:09.880805 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 111366us
I20260929 14:12:09.861464 7276 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:09.861047 7440 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:09.860059 7149 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:09.861387 7431 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:09.866245 7481 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:09.867549 7205 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:09.871619 7416 raft_consensus.cc:3032] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:09.895470 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:09.897188 5197 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:09.897842 5165 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:09.900131 5242 leader_election.cc:334] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.740s (SENT)
I20260929 14:12:09.899093 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:09.903249 5242 leader_election.cc:334] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.761s (SENT)
I20260929 14:12:09.899274 5165 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:09.903800 5242 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.905665 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:09.906263 5907 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:09.874018 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:09.859606 7267 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:09.907011 5242 leader_election.cc:334] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.859s (SENT)
I20260929 14:12:09.916690 5242 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.861657 7313 consensus_queue.cc:1035] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260929 14:12:09.880491 7476 raft_consensus.cc:2781] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:12:09.879670 7209 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:09.919158 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:09.927683 5903 raft_consensus.cc:2379] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 4: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:12:09.890345 7326 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:09.894682 7431 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:09.895615 7276 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:09.912479 6642 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:09.895301 7416 raft_consensus.cc:729] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:09.948587 7416 consensus_queue.cc:260] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:09.949968 7416 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:09.892872 7415 raft_consensus.cc:2726] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:09.911944 7406 raft_consensus.cc:2726] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:09.932725 7318 consensus_queue.cc:237] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:09.938093 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 57528.3us
I20260929 14:12:09.925364 7459 raft_consensus.cc:2726] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:09.942071 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:09.933970 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 78370.3us
I20260929 14:12:09.942338 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:09.863890 7469 consensus_queue.cc:237] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:09.906667 6922 consensus_queue.cc:1035] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.040s
I20260929 14:12:09.947247 7320 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:09.952221 7276 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:09.952618 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.966022 5346 raft_consensus.cc:2454] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:12:09.967046 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.926471 7476 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 14:12:09.960618 5242 leader_election.cc:334] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.867s (SENT)
I20260929 14:12:09.970263 5242 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.961747 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:09.931056 7491 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:09.931583 7209 raft_consensus.cc:3037] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:09.963601 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:09.954706 7326 raft_consensus.cc:2726] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:09.967619 5870 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:09.932173 7489 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:09.907030 7205 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:09.991065 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:09.971468 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:09.991979 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:09.972520 5200 raft_consensus.cc:2354] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 5.
I20260929 14:12:09.973094 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.992821 5195 raft_consensus.cc:2354] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:09.974292 5270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 234) took 3223 ms. Trace:
I20260929 14:12:09.974152 5239 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:09.993911 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:09.994608 5346 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:09.975668 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41626 (request call id 243) took 2730 ms. Trace:
I20260929 14:12:09.977181 7276 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:09.977705 5276 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:50776
I20260929 14:12:09.978991 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.938159 5907 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:09.983880 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:09.989825 5903 raft_consensus.cc:2379] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 6: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:12:10.013325 5201 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:12:09.907353 7481 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:09.981030 7408 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:09.982978 7459 raft_consensus.cc:2726] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:12:09.997951 5267 raft_consensus.cc:1261] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:12:09.992030 5268 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:12:10.024271 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:10.024274 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:10.020504 7406 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:10.028817 7481 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:10.029989 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:09.985785 7411 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:09.995728 5271 rpcz_store.cc:276] 0929 14:12:07.245172 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:07.245933 (+ 761us) service_pool.cc:227] Handling call
0929 14:12:07.246292 (+ 359us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:07.906325 (+660033us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:07.906360 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:07.930092 (+ 23732us) log.cc:842] Serialized 26 byte log entry
0929 14:12:08.305992 (+375900us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:08.306048 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:09.975295 (+1669247us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b54001090d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:09.975340 (+ 45us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:09.975383 (+ 43us) raft_consensus.cc:1671] finished
0929 14:12:09.975396 (+ 13us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:09.975601 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21465,"raft.queue_time_us":351351,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":40657,"spinlock_wait_cycles":988410752,"thread_start_us":605197,"threads_started":2,"wal-append.queue_time_us":712707}
I20260929 14:12:09.993945 5270 rpcz_store.cc:276] 0929 14:12:06.750409 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:06.750683 (+ 274us) service_pool.cc:227] Handling call
0929 14:12:06.751034 (+ 351us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:06.754818 (+ 3784us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:06.754849 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:06.768974 (+ 14125us) log.cc:842] Serialized 26 byte log entry
0929 14:12:09.968821 (+3199847us) spinlock_profiling.cc:243] Waited 2.26 ms on lock 0x7b1800081ee0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda46fc82c 00007fbda1b279dc 00007fbdaa95ea39 00007fbda1b448bb 00007fbda1b42cbf 00007fbda1b44e22 00007fbda46ee5bb 00007fbda46f4063 00007fbda46f4411
0929 14:12:09.968965 (+ 144us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:09.969000 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:09.969065 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:09.969106 (+ 41us) raft_consensus.cc:1671] finished
0929 14:12:09.969120 (+ 14us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:09.974207 (+ 5087us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11050,"raft.queue_time_us":95381,"raft.run_cpu_time_us":6782,"raft.run_wall_time_us":101921,"spinlock_wait_cycles":5197056,"thread_start_us":128260,"threads_started":1,"wal-append.queue_time_us":277964,"wal-append.run_cpu_time_us":1883,"wal-append.run_wall_time_us":1008010}
I20260929 14:12:10.000579 5276 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:10.004515 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:10.004621 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:10.033663 5276 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:10.033840 5272 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:10.004722 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:10.006044 5200 raft_consensus.cc:1226] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:10.006497 7205 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:09.998382 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:10.009461 5197 raft_consensus.cc:2354] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:09.983139 7476 raft_consensus.cc:3037] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:10.015642 5242 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:09.996922 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:10.020900 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:12:10.020756 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 53210.3us
I20260929 14:12:10.056704 5201 raft_consensus.cc:2354] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 5.
I20260929 14:12:10.036942 5898 raft_consensus.cc:2454] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:10.042161 5276 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:10.042483 6717 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:10.046536 5196 raft_consensus.cc:2396] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260929 14:12:09.997859 7489 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:10.057613 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:10.057768 5274 raft_consensus.cc:1226] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:10.064244 5201 raft_consensus.cc:2354] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:10.061187 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:10.056550 5273 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
W20260929 14:12:10.035506 7486 log.cc:924] Time spent T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.190s user 0.001s sys 0.000s
I20260929 14:12:10.064857 5276 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:10.065917 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:10.073849 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:10.074680 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:10.072554 7302 raft_consensus.cc:2781] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 14:12:10.076524 7302 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:10.069417 6717 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:10.076946 7302 raft_consensus.cc:3037] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 1 FOLLOWER]: Advancing to term 2
I20260929 14:12:10.032265 7408 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:10.050782 7297 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:10.082691 7297 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:09.999475 7041 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:10.087083 7041 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:10.050486 7438 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:10.094390 7461 consensus_queue.cc:1035] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:12:10.095135 7273 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:10.095870 7267 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:10.093286 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:10.098695 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:10.100453 6717 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:10.100747 7302 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:10.103626 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:10.104118 5351 raft_consensus.cc:1226] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:10.107080 7489 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:10.107828 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:10.108170 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:10.111086 7445 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:10.111670 7445 raft_consensus.cc:686] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:10.112190 7277 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:10.040963 7464 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:10.112561 7445 consensus_queue.cc:237] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:10.114168 5269 raft_consensus.cc:2379] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e in current term 6: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:10.113847 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:10.115051 7464 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:10.093597 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 78084.7us
I20260929 14:12:10.115937 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:10.103178 7428 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:10.118846 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
W20260929 14:12:10.118038 5242 leader_election.cc:334] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.997s (SENT)
I20260929 14:12:10.119839 5907 raft_consensus.cc:2379] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 6: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:12:10.120083 5242 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:10.116604 5345 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
W20260929 14:12:10.124243 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 256850us
W20260929 14:12:10.135260 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 171677us
W20260929 14:12:11.061825 7491 consensus_meta.cc:287] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 1.081s user 0.006s sys 0.005s
W20260929 14:12:10.178264 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 135357us
I20260929 14:12:11.108724 7491 raft_consensus.cc:2726] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:11.109020 5314 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:10.171720 5200 raft_consensus.cc:1226] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5. Ops: []
W20260929 14:12:10.933619 7452 consensus_meta.cc:287] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 1.040s user 0.011s sys 0.000s
I20260929 14:12:10.175482 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.110541 7452 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:11.110877 5200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1433) took 1096 ms. Trace:
I20260929 14:12:11.094485 7379 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:11.111686 5347 raft_consensus.cc:2379] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:11.111371 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.111842 7379 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:11.071588 7432 consensus_meta.cc:287] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 1.200s user 0.009s sys 0.004s
I20260929 14:12:10.173275 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:10.085114 7297 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:11.113327 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.113641 7432 raft_consensus.cc:2726] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:11.117933 5200 rpcz_store.cc:276] 0929 14:12:10.014869 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:10.117596 (+102727us) service_pool.cc:227] Handling call
0929 14:12:10.171539 (+ 53943us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:11.110535 (+938996us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:11.110805 (+ 270us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260929 14:12:10.185676 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 60383.5us
I20260929 14:12:11.114466 7379 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:10.310662 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:10.174665 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:10.214085 7321 consensus_queue.cc:1035] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.902s
W20260929 14:12:10.174330 7387 log.cc:924] Time spent T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260929 14:12:10.699846 7428 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:10.343822 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:10.174129 7431 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:10.412704 7302 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:10.890611 7506 log.cc:924] Time spent T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.543s user 0.000s sys 0.001s
W20260929 14:12:10.180928 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 62920.2us
I20260929 14:12:10.136672 5164 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:11.222538 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 1.08587e+06us
I20260929 14:12:11.122413 7491 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:11.112558 7452 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:10.787750 5271 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.694s user 0.010s sys 0.025s
I20260929 14:12:10.141019 5346 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 6.
W20260929 14:12:11.224817 5271 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.131s user 0.012s sys 0.031s
I20260929 14:12:10.185635 7438 raft_consensus.cc:2726] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:12:10.417740 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:11.102929 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 931774us
I20260929 14:12:11.126569 5345 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38536 (request call id 218) took 1069 ms. Trace:
W20260929 14:12:10.883872 7209 consensus_meta.cc:287] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.885s user 0.005s sys 0.005s
W20260929 14:12:10.948499 7416 consensus_meta.cc:287] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.998s user 0.008s sys 0.003s
I20260929 14:12:11.157555 5196 raft_consensus.cc:2354] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
W20260929 14:12:10.429307 7492 log.cc:924] Time spent T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.182s user 0.000s sys 0.000s
W20260929 14:12:10.816237 7460 consensus_meta.cc:287] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 1.028s user 0.002s sys 0.010s
I20260929 14:12:11.159682 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:10.765281 6917 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:12:11.217972 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 493012us
I20260929 14:12:11.218585 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:12:11.221310 5242 leader_election.cc:334] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.016s (SENT)
W20260929 14:12:10.334856 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 206719us
I20260929 14:12:11.157938 7445 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:11.225701 5345 rpcz_store.cc:276] 0929 14:12:10.057312 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:10.158582 (+101270us) service_pool.cc:227] Handling call
0929 14:12:11.126490 (+967908us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:11.226387 5201 raft_consensus.cc:2396] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260929 14:12:11.214251 5193 raft_consensus.cc:3037] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:11.228376 5346 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38536 (request call id 215) took 1376 ms. Trace:
I20260929 14:12:11.228598 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52188 (request call id 1440) took 1007 ms. Trace:
I20260929 14:12:11.226589 7432 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:11.285002 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:11.299535 5242 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:11.299443 5196 rpcz_store.cc:276] 0929 14:12:10.221062 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:10.310309 (+ 89247us) service_pool.cc:227] Handling call
0929 14:12:11.228543 (+918234us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:11.299893 5346 rpcz_store.cc:276] 0929 14:12:09.849097 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:09.993674 (+144577us) service_pool.cc:227] Handling call
0929 14:12:11.224963 (+1231289us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:11.228135 7383 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:11.235762 5200 raft_consensus.cc:2354] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
W20260929 14:12:11.298606 7506 log.cc:924] Time spent T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
W20260929 14:12:11.302815 5873 leader_election.cc:334] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.537s (SENT)
I20260929 14:12:11.296068 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.313599 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:11.224641 7469 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:11.316068 7469 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:11.319747 7469 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:11.304322 7445 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:11.304762 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.304333 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.306241 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.322858 5201 raft_consensus.cc:2454] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:11.323108 5346 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47252 (request call id 1895) took 1301 ms. Trace:
I20260929 14:12:11.323493 5346 rpcz_store.cc:276] 0929 14:12:10.021785 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:11.305989 (+1284204us) service_pool.cc:227] Handling call
0929 14:12:11.323050 (+ 17061us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:11.323661 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.324288 5201 raft_consensus.cc:3037] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:11.324501 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:11.324899 5193 raft_consensus.cc:2454] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
W20260929 14:12:11.225996 7318 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.013s sys 0.034s
I20260929 14:12:11.233940 7452 raft_consensus.cc:2781] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:11.326090 7452 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:11.326336 7438 raft_consensus.cc:2726] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:12:11.326844 7452 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:11.327366 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.303313 7432 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:11.227666 7491 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:11.234553 7460 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:11.328146 5348 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1555) took 2034 ms (client timeout 1993 ms). Trace:
I20260929 14:12:11.325099 5346 raft_consensus.cc:3032] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:11.233114 5343 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1566) took 1115 ms. Trace:
W20260929 14:12:11.325816 5193 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52188 (request call id 1438) took 1232 ms (client timeout 1575 ms). Trace:
W20260929 14:12:11.329154 5348 rpcz_store.cc:269] 0929 14:12:09.293719 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:09.339188 (+ 45469us) service_pool.cc:227] Handling call
0929 14:12:11.327223 (+1988035us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b5800183998. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:12:11.328076 (+ 853us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":34916736}
W20260929 14:12:11.330899 5193 rpcz_store.cc:269] 0929 14:12:10.093067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:10.093277 (+ 210us) service_pool.cc:227] Handling call
0929 14:12:11.325386 (+1232109us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:11.331900 5196 raft_consensus.cc:2396] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260929 14:12:11.332062 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.332823 5193 raft_consensus.cc:2354] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:11.329313 5346 raft_consensus.cc:729] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
W20260929 14:12:11.333410 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 112187us
I20260929 14:12:11.230999 7209 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:11.334702 5242 leader_election.cc:334] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.022s (SENT)
I20260929 14:12:11.335944 5242 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:11.338430 5345 raft_consensus.cc:2454] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:12:11.339434 5201 raft_consensus.cc:2454] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:12:11.339775 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.340077 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.340400 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.340880 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.341741 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.267293 6917 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:11.343163 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:11.342295 5196 raft_consensus.cc:2362] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3. Re-sending same reply.
W20260929 14:12:11.328663 7318 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.756s user 0.013s sys 0.035s
I20260929 14:12:11.344940 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.334200 5346 consensus_queue.cc:260] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:11.346649 5193 raft_consensus.cc:2354] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 5.
I20260929 14:12:11.343935 5348 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38536 (request call id 216) took 1488 ms. Trace:
I20260929 14:12:11.347654 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.355293 5348 rpcz_store.cc:276] 0929 14:12:09.855207 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:11.339475 (+1484268us) service_pool.cc:227] Handling call
0929 14:12:11.343874 (+ 4399us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:11.355633 5201 raft_consensus.cc:2454] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:11.330637 5343 rpcz_store.cc:276] 0929 14:12:10.115322 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:10.174857 (+ 59535us) service_pool.cc:227] Handling call
0929 14:12:11.230865 (+1056008us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:11.351218 6917 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:11.353614 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.328603 7407 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:11.356890 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.333388 7432 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:11.335100 7383 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:11.358624 5196 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:11.366660 5907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:11.373080 5201 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
W20260929 14:12:11.373309 5242 leader_election.cc:334] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.053s (SENT)
W20260929 14:12:11.374083 5242 leader_election.cc:334] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.065s (SENT)
I20260929 14:12:11.374046 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.374568 5242 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
W20260929 14:12:11.367241 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 212644us
I20260929 14:12:11.367555 7438 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:11.366743 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.376662 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.378059 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.380280 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.380838 5193 raft_consensus.cc:2354] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:11.380682 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.381624 5196 raft_consensus.cc:2396] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260929 14:12:11.381645 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.382467 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.386415 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.387329 5196 raft_consensus.cc:2354] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:11.375885 7383 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:11.388139 5193 raft_consensus.cc:2396] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260929 14:12:11.388718 5201 raft_consensus.cc:2354] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 5.
I20260929 14:12:11.389766 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.391059 5201 raft_consensus.cc:2396] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260929 14:12:11.391969 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.378958 7438 raft_consensus.cc:2726] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:11.399788 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.372381 7407 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:11.400470 7383 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:11.400647 5342 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:38536 (request call id 210) took 2059 ms (client timeout 1860 ms). Trace:
I20260929 14:12:11.400765 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.400050 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.401389 5196 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
W20260929 14:12:11.401146 5342 rpcz_store.cc:269] 0929 14:12:09.341610 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:09.343103 (+ 1493us) service_pool.cc:227] Handling call
0929 14:12:11.399632 (+2056529us) spinlock_profiling.cc:243] Waited 30.2 ms on lock 0x7b5800183998. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:12:11.400579 (+ 947us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":69386496}
I20260929 14:12:11.402407 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.403373 5196 raft_consensus.cc:2396] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260929 14:12:11.401966 5200 raft_consensus.cc:2362] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3. Re-sending same reply.
I20260929 14:12:11.399832 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.403684 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.386543 7445 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:11.405455 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.406075 5200 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
W20260929 14:12:11.407233 7507 log.cc:924] Time spent T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20260929 14:12:11.408381 7469 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:11.409210 7469 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:11.403656 6917 consensus_queue.cc:1035] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.007s
W20260929 14:12:11.409795 5269 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41626 (request call id 258) took 1594 ms (client timeout 2017 ms). Trace:
W20260929 14:12:11.412292 5269 rpcz_store.cc:269] 0929 14:12:09.815065 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:09.815251 (+ 186us) service_pool.cc:227] Handling call
0929 14:12:11.409593 (+1594342us) spinlock_profiling.cc:243] Waited 254 ms on lock 0x7b5400109ad4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:11.409739 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":584180608}
I20260929 14:12:11.386300 7431 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:11.414368 7460 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:11.411919 7476 consensus_meta.cc:287] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 1.341s user 0.010s sys 0.003s
I20260929 14:12:11.415309 7476 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:11.419094 7476 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:11.419826 5198 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52182 (request call id 1573) took 3163 ms (client timeout 2240 ms). Trace:
W20260929 14:12:11.420249 5198 rpcz_store.cc:269] 0929 14:12:08.256140 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:08.256245 (+ 105us) service_pool.cc:227] Handling call
0929 14:12:11.419613 (+3163368us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54001464d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:11.419726 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":457481600}
I20260929 14:12:11.422767 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:11.423318 5903 raft_consensus.cc:3037] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:12:11.423674 5350 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1548) took 2188 ms (client timeout 1660 ms). Trace:
W20260929 14:12:11.424050 5350 rpcz_store.cc:269] 0929 14:12:09.235464 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:09.235583 (+ 119us) service_pool.cc:227] Handling call
0929 14:12:11.423468 (+2187885us) spinlock_profiling.cc:243] Waited 244 ms on lock 0x7b54003f70d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:11.423607 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":560689536}
I20260929 14:12:11.425822 5350 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:47246 (request call id 547) took 2017 ms. Trace:
I20260929 14:12:11.437409 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.426946 7412 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:11.439487 7412 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:11.428540 5194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1589) took 2023 ms. Trace:
I20260929 14:12:11.425983 7407 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:11.430343 7383 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:11.430603 7459 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:11.430941 5903 raft_consensus.cc:2454] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:12:11.432817 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:11.419178 7452 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:11.438200 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.438854 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.437693 5350 rpcz_store.cc:276] 0929 14:12:09.408576 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:11.424726 (+2016150us) service_pool.cc:227] Handling call
0929 14:12:11.425748 (+ 1022us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:11.442035 5201 raft_consensus.cc:2362] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5. Re-sending same reply.
I20260929 14:12:11.429553 7209 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:11.440915 5194 rpcz_store.cc:276] 0929 14:12:09.404966 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:09.405136 (+ 170us) service_pool.cc:227] Handling call
0929 14:12:09.405482 (+ 346us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:09.450507 (+ 45025us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:09.450548 (+ 41us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:09.450596 (+ 48us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:09.450633 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:11.428341 (+1977708us) spinlock_profiling.cc:243] Waited 44.8 ms on lock 0x7b5400351454. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:11.428382 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:11.428486 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":103024512}
I20260929 14:12:11.495862 5196 raft_consensus.cc:2354] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:11.489586 5167 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:12:11.435428 7408 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:11.490188 5316 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:11.493911 5193 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:12:11.447327 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.496786 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.497372 5198 raft_consensus.cc:2354] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:11.497609 5196 raft_consensus.cc:2354] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:11.497547 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.498561 5193 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:11.498721 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.499562 5196 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:12:11.499634 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:12:11.501729 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 392777us
I20260929 14:12:11.502703 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.503362 5198 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:11.503906 5350 raft_consensus.cc:1226] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:11.506253 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.506932 5350 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 6.
I20260929 14:12:11.508093 5348 raft_consensus.cc:1226] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: []
W20260929 14:12:11.508263 5873 leader_election.cc:334] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.090s (SENT)
I20260929 14:12:11.509984 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.510876 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.488255 7445 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:11.511478 5201 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:12:11.513340 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.492120 7476 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:11.523886 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1613) took 1479 ms. Trace:
I20260929 14:12:11.520567 5194 raft_consensus.cc:2354] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:11.506309 7209 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:11.507581 7494 raft_consensus.cc:2781] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:12:11.506769 6917 raft_consensus.cc:2781] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:11.493381 7383 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:11.452885 7518 consensus_queue.cc:1035] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.119s
I20260929 14:12:11.518054 7408 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:11.524611 5271 rpcz_store.cc:276] 0929 14:12:10.044057 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:10.044519 (+ 462us) service_pool.cc:227] Handling call
0929 14:12:10.044825 (+ 306us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:10.045027 (+ 202us) raft_consensus.cc:1520] Early marking committed up to index 4
0929 14:12:10.071292 (+ 26265us) log.cc:842] Serialized 14 byte log entry
0929 14:12:11.225239 (+1153947us) log.cc:842] Serialized 14 byte log entry
0929 14:12:11.522600 (+297361us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:11.522685 (+ 85us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:12:11.522760 (+ 75us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:11.522888 (+ 128us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:11.523815 (+ 927us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22218,"prepare.queue_time_us":38335,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":73,"raft.queue_time_us":23008,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":1428117,"threads_started":2}
I20260929 14:12:11.530342 5901 raft_consensus.cc:2379] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 6: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:12:11.533924 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1597) took 1634 ms. Trace:
I20260929 14:12:11.575886 5199 rpcz_store.cc:276] 0929 14:12:09.899078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:09.899462 (+ 384us) service_pool.cc:227] Handling call
0929 14:12:09.899762 (+ 300us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:09.913764 (+ 14002us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:09.913807 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:09.921148 (+ 7341us) log.cc:842] Serialized 26 byte log entry
0929 14:12:10.158349 (+237201us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:10.158386 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:10.158466 (+ 80us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:11.533631 (+1375165us) raft_consensus.cc:1671] finished
0929 14:12:11.533662 (+ 31us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:11.533847 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":11606,"raft.queue_time_us":30593,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":23259,"thread_start_us":236767,"threads_started":1,"wal-append.queue_time_us":1302241}
I20260929 14:12:11.495059 7459 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:11.522876 7469 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:11.580163 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 206089us
I20260929 14:12:11.581609 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:11.586097 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.586723 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:11.589392 5242 leader_election.cc:334] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.090s (SENT)
I20260929 14:12:11.589391 5194 raft_consensus.cc:2396] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260929 14:12:11.590221 5270 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:12:11.590595 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.592388 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:11.592967 5350 raft_consensus.cc:3037] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:11.596220 5870 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:11.596992 5242 leader_election.cc:334] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.093s (SENT)
I20260929 14:12:11.598078 5242 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:11.524978 7445 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:11.545392 6917 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:11.597137 5199 raft_consensus.cc:2354] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:11.579197 7438 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:11.579744 7491 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:11.538204 7476 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:11.596767 7469 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:11.617795 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.618460 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.544080 7209 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:11.601162 7412 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:11.618934 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 98229.2us
W20260929 14:12:11.600843 5901 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45252 (request call id 454) took 3337 ms (client timeout 1550 ms). Trace:
I20260929 14:12:11.621837 5197 raft_consensus.cc:2354] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
W20260929 14:12:11.660785 5197 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52188 (request call id 1434) took 1643 ms (client timeout 2053 ms). Trace:
I20260929 14:12:11.662587 5199 raft_consensus.cc:2396] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260929 14:12:11.622530 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:11.664717 5349 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1553) took 2405 ms (client timeout 1562 ms). Trace:
W20260929 14:12:11.665196 5349 rpcz_store.cc:269] 0929 14:12:09.259556 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:09.318188 (+ 58632us) service_pool.cc:227] Handling call
0929 14:12:11.622376 (+2304188us) spinlock_profiling.cc:243] Waited 22.5 ms on lock 0x7b5800183998. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:12:11.664625 (+ 42249us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51712896}
W20260929 14:12:11.667852 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 71727.9us
I20260929 14:12:11.671132 5344 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47258 (request call id 1556) took 2337 ms. Trace:
I20260929 14:12:11.671211 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.671866 5200 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:11.671622 5344 rpcz_store.cc:276] 0929 14:12:09.333536 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:09.516410 (+182874us) service_pool.cc:227] Handling call
0929 14:12:09.516841 (+ 431us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:09.546593 (+ 29752us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:09.546636 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:09.546692 (+ 56us) raft_consensus.cc:1631] Marking committed up to 7
0929 14:12:09.547011 (+ 319us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:11.670927 (+2123916us) spinlock_profiling.cc:243] Waited 29.4 ms on lock 0x7b5400258fd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:11.670973 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:11.671081 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":67692928}
W20260929 14:12:11.672935 7508 log.cc:924] Time spent T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.447s user 0.000s sys 0.001s
I20260929 14:12:11.673738 7452 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:11.674701 5272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1611) took 1631 ms. Trace:
I20260929 14:12:11.675091 5272 rpcz_store.cc:276] 0929 14:12:10.043475 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:10.045545 (+ 2070us) service_pool.cc:227] Handling call
0929 14:12:10.070674 (+ 25129us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:10.071105 (+ 431us) raft_consensus.cc:1520] Early marking committed up to index 2
0929 14:12:10.079316 (+ 8211us) log.cc:842] Serialized 14 byte log entry
0929 14:12:10.211689 (+132373us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:11.123073 (+911384us) log.cc:842] Serialized 26 byte log entry
0929 14:12:11.156470 (+ 33397us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:11.156518 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:11.156590 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:11.430414 (+273824us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b18003c8c80. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b45764 00007fbda1b48b11 00007fbda1b48ac9 00007fbda1b48a59 00007fbda1b48a21 00007fbda1b47d1d 00007fbdaada9204 00007fbdaada9039 00007fbda1b27dc6
0929 14:12:11.672388 (+241974us) spinlock_profiling.cc:243] Waited 4.11 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda46efbee 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551 00007fbda46f8509 00007fbda46f8499
0929 14:12:11.674463 (+ 2075us) raft_consensus.cc:1671] finished
0929 14:12:11.674485 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:11.674603 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51938,"prepare.queue_time_us":20644,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":14471,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":10156,"spinlock_wait_cycles":52783104,"thread_start_us":120898,"threads_started":1,"wal-append.queue_time_us":1037939}
I20260929 14:12:11.677610 5870 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:11.678692 5870 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:11.679195 7524 log.cc:924] Time spent T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260929 14:12:11.680109 7376 raft_consensus.cc:2741] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260929 14:12:11.680647 7383 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:11.680709 7376 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:12:11.681294 7383 raft_consensus.cc:2726] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:11.682482 5870 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:11.670140 7495 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:11.683238 7459 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:11.684309 7495 raft_consensus.cc:686] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 6 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:11.685528 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.686219 7445 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:11.686451 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.679111 7526 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:11.687434 7495 consensus_queue.cc:237] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:11.685101 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.689540 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.613533 6917 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
W20260929 14:12:11.737882 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 159359us
I20260929 14:12:11.694833 5196 raft_consensus.cc:2379] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 7: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:11.673132 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.696063 7476 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:11.661388 5197 rpcz_store.cc:269] 0929 14:12:10.017149 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:10.060827 (+ 43678us) service_pool.cc:227] Handling call
0929 14:12:11.621495 (+1560668us) spinlock_profiling.cc:243] Waited 7.28 ms on lock 0x7b540023b328. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9e6434 00007fbda99af437 00007fbda99aebc8 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:11.660711 (+ 39216us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":16735488}
I20260929 14:12:11.697189 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.689431 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.741410 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.741582 5200 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
W20260929 14:12:11.667744 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 145628us
I20260929 14:12:11.687834 7526 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:11.705173 5201 raft_consensus.cc:2396] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260929 14:12:11.705564 5193 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:12:11.587574 7408 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:11.741655 5199 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:11.757706 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.758736 5199 raft_consensus.cc:2396] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260929 14:12:11.748353 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 207062us
I20260929 14:12:11.754796 5316 leader_election.cc:398] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6.
I20260929 14:12:11.759423 5316 leader_election.cc:401] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:12:11.761041 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.761363 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.761997 5197 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:12:11.762801 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.763365 5197 raft_consensus.cc:2454] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:12:11.766476 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:11.624367 7491 raft_consensus.cc:465] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Not starting pre-election -- already a leader
I20260929 14:12:11.746845 5871 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:11.777066 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.780179 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.780522 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.770296 5314 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:11.782723 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.785805 5164 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:11.785488 5193 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:11.747841 7376 raft_consensus.cc:2781] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:11.615939 7438 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:11.792076 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.790004 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.799585 5201 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:11.632196 7469 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:11.800110 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:11.802683 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 55948us
I20260929 14:12:11.811007 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.800640 5197 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:12:11.828549 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.756582 7494 raft_consensus.cc:686] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:11.769583 7519 raft_consensus.cc:2726] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6.
I20260929 14:12:11.831351 5199 raft_consensus.cc:2454] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:12:11.828272 5193 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:11.832280 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.832384 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:11.838620 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.798614 7520 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:11.630628 7523 consensus_queue.cc:1046] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.179s
I20260929 14:12:11.830637 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:11.834044 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:11.835747 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.793047 7491 raft_consensus.cc:2726] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:11.839128 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.847270 5273 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:11.832796 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.839890 5198 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:11.848361 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:11.849383 5906 raft_consensus.cc:2379] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 4: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
W20260929 14:12:11.840101 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 243188us
I20260929 14:12:11.850229 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:11.843143 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.845301 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.838598 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:12:11.706472 5901 rpcz_store.cc:269] 0929 14:12:08.261136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:08.262533 (+ 1397us) service_pool.cc:227] Handling call
0929 14:12:11.432649 (+3170116us) spinlock_profiling.cc:243] Waited 6.19 ms on lock 0x7b5800027698. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:12:11.596158 (+163509us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14241024}
I20260929 14:12:11.844718 5199 raft_consensus.cc:2379] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 4: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:11.803401 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:11.886986 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.852321 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.888373 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:11.895202 5242 connection.cc:232] RPC call timeout handler was delayed by -1.7038858210000853s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1885s for a response
I20260929 14:12:11.887364 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.895813 5201 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:11.896739 5200 raft_consensus.cc:2454] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:12:11.897256 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:11.899264 5273 raft_consensus.cc:2379] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 4: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:11.899873 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.901834 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:11.904117 5350 raft_consensus.cc:2454] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
W20260929 14:12:11.905258 5242 leader_election.cc:334] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.146s (SENT)
I20260929 14:12:11.905161 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.906499 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.907159 5197 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:12:11.905097 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.905874 5242 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:11.907720 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:11.908274 5350 raft_consensus.cc:3037] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:11.909287 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.910590 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.911137 5199 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:11.843583 7520 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:11.906208 5870 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:11.886477 7209 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:11.882385 7335 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:11.935110 7209 raft_consensus.cc:2781] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:11.913806 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:11.914502 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.841629 7421 raft_consensus.cc:2726] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:11.920295 7534 log.cc:924] Time spent T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
I20260929 14:12:11.921236 5900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:11.922968 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:11.924342 7469 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:11.929913 7440 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260929 14:12:11.930266 5242 leader_election.cc:334] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.728s (SENT)
I20260929 14:12:11.839390 5905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:11.887523 7438 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:11.935544 7335 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:12.085984 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:12.079679 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:12.003744 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.091655 5275 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:11.937386 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.091951 5193 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:11.936168 5200 raft_consensus.cc:2454] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:12.042835 5344 raft_consensus.cc:1261] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260929 14:12:11.987668 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:12.094519 5350 raft_consensus.cc:2454] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:12:12.096989 5348 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 1887) took 2656 ms. Trace:
I20260929 14:12:12.109170 5348 rpcz_store.cc:276] 0929 14:12:09.441032 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:11.365589 (+1924557us) service_pool.cc:227] Handling call
0929 14:12:11.368123 (+ 2534us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:11.511746 (+143623us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:12.096713 (+584967us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b5400387054. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:12.096914 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":321365376}
I20260929 14:12:12.110237 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:11.935660 7209 raft_consensus.cc:686] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [term 5 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:11.968217 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:12.101408 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 171157us
W20260929 14:12:12.114182 5242 connection.cc:232] RPC call timeout handler was delayed by -1.5995333000000755s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22130000000000002s for a response
I20260929 14:12:12.113024 5271 raft_consensus.cc:1261] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:12:12.114732 5273 raft_consensus.cc:1261] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260929 14:12:12.096050 5347 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1562) took 2282 ms (client timeout 2739 ms). Trace:
I20260929 14:12:12.117929 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:12.118597 5347 rpcz_store.cc:269] 0929 14:12:09.814073 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:09.816053 (+ 1980us) service_pool.cc:227] Handling call
0929 14:12:12.095875 (+2279822us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b540005b9d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:12.095994 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2882501376}
I20260929 14:12:12.116914 5199 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:12.084074 7376 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:12.121449 7376 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:12.117708 7499 consensus_queue.cc:1035] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.004s
W20260929 14:12:12.078771 7539 log.cc:924] Time spent T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.177s user 0.000s sys 0.000s
W20260929 14:12:12.079466 7529 log.cc:924] Time spent T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.279s user 0.001s sys 0.000s
I20260929 14:12:12.122975 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.138793 5200 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 7.
I20260929 14:12:12.140113 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.141744 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:11.936820 7520 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:11.906940 5343 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 217) took 2046 ms. Trace:
I20260929 14:12:12.031780 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:12.125696 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:12.150344 5269 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:12.118944 5193 raft_consensus.cc:2454] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
W20260929 14:12:11.986783 7538 log.cc:924] Time spent T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
I20260929 14:12:12.143330 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.095305 7523 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:12.096629 7542 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:12.146556 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:12.158489 5342 raft_consensus.cc:2379] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 3: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:12.146414 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.146812 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:12.134593 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:12.158641 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:12.160006 5199 raft_consensus.cc:2396] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260929 14:12:12.160163 5193 raft_consensus.cc:3037] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:12.148494 5239 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.154035 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.163172 7440 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:12:12.125700 7469 consensus_queue.cc:1046] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.030s
I20260929 14:12:12.163563 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.164381 7469 raft_consensus.cc:3032] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:12:12.164757 7469 raft_consensus.cc:729] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 3 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:12.165571 7469 consensus_queue.cc:260] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:12.166551 7335 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:12.166818 7469 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:12.170337 7523 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:12.172338 7523 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:12.173296 7542 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:12.173554 7209 consensus_queue.cc:237] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:12.175719 7542 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:12.147718 5343 rpcz_store.cc:276] 0929 14:12:09.857561 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:11.367432 (+1509871us) service_pool.cc:227] Handling call
0929 14:12:11.367750 (+ 318us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:11.372028 (+ 4278us) raft_consensus.cc:1520] Early marking committed up to index 6
0929 14:12:11.530105 (+158077us) log.cc:842] Serialized 14 byte log entry
0929 14:12:11.843333 (+313228us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:11.846510 (+ 3177us) raft_consensus.cc:1631] Marking committed up to 6
0929 14:12:11.850476 (+ 3966us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:11.888195 (+ 37719us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:11.903525 (+ 15330us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34100,"prepare.queue_time_us":232235,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":938,"raft.queue_time_us":40035,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":6495,"thread_start_us":385993,"threads_started":2,"wal-append.queue_time_us":381404}
W20260929 14:12:12.180564 7432 consensus_meta.cc:287] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.804s user 0.011s sys 0.000s
I20260929 14:12:12.180974 7432 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:12.182472 5195 raft_consensus.cc:2379] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 6: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:12.183003 7432 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:12.183866 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.184500 5199 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:12:12.185097 5907 raft_consensus.cc:1226] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:12.184796 6917 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:12.186798 7491 consensus_queue.cc:1046] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:12:12.186800 6917 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:12.187788 5192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1602) took 2287 ms. Trace:
I20260929 14:12:12.188196 5192 rpcz_store.cc:276] 0929 14:12:09.900645 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:09.901192 (+ 547us) service_pool.cc:227] Handling call
0929 14:12:09.902472 (+ 1280us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:10.005461 (+102989us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:10.005488 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:10.005528 (+ 40us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:12:10.005556 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:12.187579 (+2182023us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b540031d654. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:12.187620 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:12.187736 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":236449024}
I20260929 14:12:12.189688 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.184724 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:12.192420 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:12.165112 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.193415 5342 raft_consensus.cc:2379] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:11.967803 7526 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:12.159109 5201 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 7.
I20260929 14:12:12.188890 5270 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:12.189913 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:12.197366 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:12.198191 5192 raft_consensus.cc:3037] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:12.200098 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:12.156594 5200 raft_consensus.cc:2454] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
W20260929 14:12:12.206075 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 91470.2us
W20260929 14:12:12.206880 5242 connection.cc:232] RPC call timeout handler was delayed by -1.521466190999945s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2068s for a response
W20260929 14:12:12.207178 5242 connection.cc:232] RPC call timeout handler was delayed by -1.5131194670000241s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15980000000000003s for a response
W20260929 14:12:12.207459 5242 connection.cc:232] RPC call timeout handler was delayed by -1.496933300000137s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23270000000000002s for a response
W20260929 14:12:12.207931 5242 connection.cc:232] RPC call timeout handler was delayed by -1.2327931250001711s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2207s for a response
W20260929 14:12:12.208379 5242 connection.cc:232] RPC call timeout handler was delayed by -1.2057931250001275s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2237s for a response
W20260929 14:12:12.208742 5242 connection.cc:232] RPC call timeout handler was delayed by -1.1717591920000814s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19930000000000003s for a response
W20260929 14:12:12.209038 5242 connection.cc:232] RPC call timeout handler was delayed by -1.1265516280000156s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
I20260929 14:12:12.205192 5269 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:12.213511 5270 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:12.214098 5347 raft_consensus.cc:1226] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:12.209647 5242 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.217692 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:12.219705 7406 raft_consensus.cc:2726] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260929 14:12:12.212543 7320 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:12.221333 7320 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:12.188994 7376 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:12.223223 7320 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:12.224000 7376 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:12.147569 5275 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:12:12.227214 5350 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:12.043643 7494 consensus_queue.cc:237] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:12.238240 5351 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 208) took 2899 ms. Trace:
I20260929 14:12:12.238806 5351 rpcz_store.cc:276] 0929 14:12:09.338528 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:09.527092 (+188564us) service_pool.cc:227] Handling call
0929 14:12:09.527385 (+ 293us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:10.104917 (+577532us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:12.237921 (+2133004us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b540027e054. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:12.238159 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1297223936}
I20260929 14:12:12.240840 7525 consensus_queue.cc:1046] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 14:12:12.241891 7525 raft_consensus.cc:3032] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:12.242218 7525 raft_consensus.cc:729] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:12.242847 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:12.242985 7525 consensus_queue.cc:260] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:12.245818 5346 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:12.244210 5274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1597) took 2259 ms. Trace:
I20260929 14:12:12.246666 5274 rpcz_store.cc:276] 0929 14:12:09.984329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:09.984465 (+ 136us) service_pool.cc:227] Handling call
0929 14:12:10.010301 (+ 25836us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:10.064359 (+ 54058us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:12.244016 (+2179657us) spinlock_profiling.cc:243] Waited 47.3 ms on lock 0x7b5400321ed4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:12.244158 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":108727040}
I20260929 14:12:12.235357 7520 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:12.258015 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:12.245116 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:12.262077 5901 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:12.263299 7520 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:12.265349 7408 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:12.268522 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 58942us
I20260929 14:12:12.271091 5242 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.273351 7526 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260929 14:12:12.279372 7516 log.cc:924] Time spent T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.960s user 0.005s sys 0.000s
W20260929 14:12:12.243968 5898 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45248 (request call id 329) took 2398 ms (client timeout 2013 ms). Trace:
W20260929 14:12:12.280786 5898 rpcz_store.cc:269] 0929 14:12:09.845553 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:09.856561 (+ 11008us) service_pool.cc:227] Handling call
0929 14:12:12.243789 (+2387228us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b540060f9d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:12.243896 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":263412992}
I20260929 14:12:12.283617 7432 raft_consensus.cc:3032] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Stepping down as leader of term 7
I20260929 14:12:12.284164 7432 raft_consensus.cc:729] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:12.285740 5268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1610) took 2278 ms. Trace:
I20260929 14:12:12.285771 5314 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:12.286859 7540 raft_consensus.cc:3032] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:12.287410 7540 raft_consensus.cc:729] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:12.289389 7430 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:12.289825 7430 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:12.290386 7430 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:12.286134 5268 rpcz_store.cc:276] 0929 14:12:10.007034 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:10.028564 (+ 21530us) service_pool.cc:227] Handling call
0929 14:12:10.028838 (+ 274us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:10.139331 (+110493us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:10.139375 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:11.105962 (+966587us) log.cc:842] Serialized 26 byte log entry
0929 14:12:11.229851 (+123889us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:12:11.229893 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:11.229974 (+ 81us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:12.278894 (+1048920us) spinlock_profiling.cc:243] Waited 9.17 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda46efbee 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551 00007fbda46f8509 00007fbda46f8499
0929 14:12:12.285460 (+ 6566us) raft_consensus.cc:1671] finished
0929 14:12:12.285479 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:12.285683 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":94764,"raft.queue_time_us":110178,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":16842,"spinlock_wait_cycles":21101312,"thread_start_us":123565,"threads_started":1,"wal-append.queue_time_us":201998}
I20260929 14:12:12.288455 7540 consensus_queue.cc:260] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:12.292092 7540 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:12.292683 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:12.262458 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:12.294943 5345 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 213) took 2952 ms. Trace:
I20260929 14:12:12.295361 5345 rpcz_store.cc:276] 0929 14:12:09.342460 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:11.348238 (+2005778us) service_pool.cc:227] Handling call
0929 14:12:11.438044 (+ 89806us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:11.438354 (+ 310us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:11.438382 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:11.438421 (+ 39us) raft_consensus.cc:1631] Marking committed up to 6
0929 14:12:11.438449 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:11.438503 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:12.294780 (+856277us) spinlock_profiling.cc:243] Waited 89.4 ms on lock 0x7b54003f70d0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6ca09 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:12.294885 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":394,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"spinlock_wait_cycles":205656960}
I20260929 14:12:12.284902 7432 consensus_queue.cc:260] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:12.298703 7432 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:12.301352 5242 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.245829 7525 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:12.302080 7464 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:12.308102 5903 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260929 14:12:12.309759 7461 consensus_queue.cc:1035] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:12:12.311378 7461 consensus_queue.cc:1035] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
W20260929 14:12:12.312937 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 88251.4us
I20260929 14:12:12.313956 5316 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:12.315722 5242 leader_election.cc:334] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.598s (SENT)
I20260929 14:12:12.316157 5242 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.320603 7464 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:12.320792 7302 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.005s user 0.005s sys 0.002s
W20260929 14:12:12.321179 7302 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.006s user 0.005s sys 0.002s
W20260929 14:12:12.322142 5242 leader_election.cc:334] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.640s (SENT)
I20260929 14:12:12.340914 5346 raft_consensus.cc:2454] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
W20260929 14:12:12.346911 5346 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1565) took 2275 ms (client timeout 2806 ms). Trace:
W20260929 14:12:12.347295 5346 rpcz_store.cc:269] 0929 14:12:10.066226 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:11.324287 (+1258061us) service_pool.cc:227] Handling call
0929 14:12:12.245662 (+921375us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b580012feb0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda49e8f63 00007fbda4a4caba 00007fbda4a643af 00007fbda4a65b4e 00007fbda4a71228 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c
0929 14:12:12.341635 (+ 95973us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1589957888}
I20260929 14:12:12.350960 7040 raft_consensus.cc:2781] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:12:12.351701 7040 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:12.352313 7040 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:12.336218 7430 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:12.370062 7430 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:12.371942 7040 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:12.374192 7040 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:12.420090 5164 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:12.422616 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.425073 5239 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:12.425814 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.322592 5242 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.424865 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.422523 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.431180 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.432320 5200 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:12.432868 5199 raft_consensus.cc:2354] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e for earlier term 7. Current term is 8.
W20260929 14:12:12.433722 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 111358us
I20260929 14:12:12.434363 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.435349 5199 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
W20260929 14:12:12.435843 5242 leader_election.cc:334] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.723s (SENT)
I20260929 14:12:12.436537 5242 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.437610 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:12:12.438838 5242 leader_election.cc:334] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.802s (SENT)
I20260929 14:12:12.439404 5242 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.443046 5193 raft_consensus.cc:2454] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:12:12.443148 5267 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1614) took 2398 ms. Trace:
I20260929 14:12:12.443711 5267 rpcz_store.cc:276] 0929 14:12:10.044354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:12.293493 (+2249139us) service_pool.cc:227] Handling call
0929 14:12:12.293862 (+ 369us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:12.294053 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 4
0929 14:12:12.294312 (+ 259us) log.cc:842] Serialized 14 byte log entry
0929 14:12:12.442460 (+148148us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:12.442555 (+ 95us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:12:12.442653 (+ 98us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:12.442761 (+ 108us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:12.442959 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":123,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"thread_start_us":147584,"threads_started":1,"wal-append.queue_time_us":69751}
I20260929 14:12:12.446533 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.447109 5192 raft_consensus.cc:2454] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:12:12.451748 5314 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:12.452478 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.455760 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.526383 5196 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:12.527071 5194 raft_consensus.cc:2396] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260929 14:12:12.527380 5197 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 7.
I20260929 14:12:12.527479 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.457398 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.354933 5316 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:12.528069 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.528726 5198 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:12.529170 7041 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election won for term 6
W20260929 14:12:12.529285 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 174066us
I20260929 14:12:12.460508 5906 raft_consensus.cc:1226] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:12.464515 7438 raft_consensus.cc:2781] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:12.532194 7438 raft_consensus.cc:686] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:12.532974 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.534052 7438 consensus_queue.cc:237] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:12.465473 7464 raft_consensus.cc:2726] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:12.465850 7406 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:12.537552 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.538862 5198 raft_consensus.cc:2396] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260929 14:12:12.539960 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.541280 7341 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:12.541949 7341 raft_consensus.cc:686] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:12.466792 7494 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:12.466878 7408 raft_consensus.cc:2726] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:12.467553 7432 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:12.467736 7412 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:12.467924 7523 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:12.475123 5346 raft_consensus.cc:1226] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:12.480367 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:12.434389 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:12.530028 7041 raft_consensus.cc:686] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:12.491058 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:12.499764 5344 raft_consensus.cc:1226] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 2. Current term is 3. Ops: []
W20260929 14:12:12.506906 5195 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52188 (request call id 1452) took 1450 ms (client timeout 1553 ms). Trace:
I20260929 14:12:12.508847 5873 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:12.549284 7412 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:12.549508 7432 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:12.511449 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:12.548714 5199 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:12.447633 5193 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:12:12.551465 5196 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:09.138517 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260929 14:12:12.521804 5242 leader_election.cc:334] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.885s (SENT)
W20260929 14:12:12.526980 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 75608.5us
I20260929 14:12:12.534065 5197 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:12.485270 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:12.548451 5198 raft_consensus.cc:1261] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:12:12.553330 5200 raft_consensus.cc:3037] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:12.552039 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:12.543745 5316 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:12:12.548790 5195 rpcz_store.cc:269] 0929 14:12:11.056400 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:11.236442 (+180042us) service_pool.cc:227] Handling call
0929 14:12:12.506431 (+1269989us) spinlock_profiling.cc:243] Waited 873 ms on lock 0x7b5400352354. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:12.506699 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2006833536}
I20260929 14:12:12.556175 5242 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.558907 5194 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:12.559837 7412 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:12.561652 5314 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:12.543809 7341 consensus_queue.cc:237] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:12.570272 5272 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 7
I20260929 14:12:12.575824 7525 raft_consensus.cc:2726] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:12.579252 5096 catalog_manager.cc:5495] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 1 to 5. New cstate: current_term: 5 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:12.582190 5096 catalog_manager.cc:5495] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 1 to 3, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 3 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260929 14:12:12.586853 7432 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:12.587740 7531 consensus_queue.cc:1046] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260929 14:12:12.588099 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:12.586572 5096 catalog_manager.cc:5495] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 1 to 2, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 2 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:12.590721 7379 consensus_queue.cc:1046] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:12.590802 5096 catalog_manager.cc:5495] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:12.593549 7495 raft_consensus.cc:3032] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:12:12.594650 7495 raft_consensus.cc:729] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:12.595738 7495 consensus_queue.cc:260] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:12.597322 7495 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:12.597841 5194 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:12.601467 5274 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:12.601832 6868 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260929 14:12:12.602008 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 58528.8us
I20260929 14:12:12.603047 5316 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:12.555014 7523 raft_consensus.cc:686] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:12.604625 7430 raft_consensus.cc:2781] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:12:12.605583 7040 raft_consensus.cc:2726] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:12.605265 7523 consensus_queue.cc:237] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:12.605098 5314 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:12.614766 7408 raft_consensus.cc:2726] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260929 14:12:12.614660 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 92944.7us
I20260929 14:12:12.632989 5196 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260929 14:12:12.605291 7430 raft_consensus.cc:686] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:12.649152 5201 raft_consensus.cc:1261] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260929 14:12:12.650403 5198 raft_consensus.cc:1261] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:12:12.649699 5201 pending_rounds.cc:76] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd: Aborting all ops after (but not including) 4
I20260929 14:12:12.653337 5201 pending_rounds.cc:97] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260929 14:12:12.653638 5201 raft_consensus.cc:2866] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260929 14:12:12.653879 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 52226.2us
I20260929 14:12:12.655313 5201 raft_consensus.cc:1226] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:12.655405 7476 consensus_queue.cc:1046] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20260929 14:12:12.656559 5242 leader_election.cc:334] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.068s (SENT)
I20260929 14:12:12.656653 7476 raft_consensus.cc:3032] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:12.657092 5242 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.657131 7476 raft_consensus.cc:729] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:12.657330 7494 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:12.658497 7041 consensus_queue.cc:237] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:12.658555 7476 consensus_queue.cc:260] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:12.659726 5242 leader_election.cc:334] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.213s (SENT)
I20260929 14:12:12.659902 7494 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:12.660323 7476 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:12.661397 7438 raft_consensus.cc:2726] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:12.660226 5242 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.662498 7438 raft_consensus.cc:2726] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260929 14:12:12.664016 5242 leader_election.cc:334] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.237s (SENT)
I20260929 14:12:12.664129 5196 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260929 14:12:12.664453 5242 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.664616 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:12.669762 5316 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:12:12.670176 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 65274.9us
I20260929 14:12:12.671618 5316 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:12.673785 7321 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:12.675285 7321 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:12.680480 7432 raft_consensus.cc:3032] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:12:12.681388 5873 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.682785 7379 raft_consensus.cc:2781] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:12.683274 7379 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:12.683386 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 72145.2us
I20260929 14:12:12.683746 7379 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:12.684396 5873 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:12.685030 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:12.685451 7525 raft_consensus.cc:2726] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:12.651883 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:12.680977 7432 raft_consensus.cc:729] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:12.703320 5200 raft_consensus.cc:2454] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:12.710182 5274 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:12.711158 5267 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:12.673085 7311 raft_consensus.cc:2781] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:12:12.722266 5201 raft_consensus.cc:1261] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260929 14:12:12.718600 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 47189us
I20260929 14:12:12.711755 7438 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:12.734824 5344 raft_consensus.cc:1261] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260929 14:12:12.735817 5192 raft_consensus.cc:1226] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:12.737462 5343 raft_consensus.cc:1226] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7. Ops: []
W20260929 14:12:12.740132 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 55907.1us
I20260929 14:12:12.689147 7430 consensus_queue.cc:237] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:12.746296 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 75065.8us
I20260929 14:12:12.754765 5316 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:12.728521 5096 catalog_manager.cc:5495] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:12.748526 7311 raft_consensus.cc:686] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:12.757652 5096 catalog_manager.cc:5495] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:12.758482 7311 consensus_queue.cc:237] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:12.759608 5096 catalog_manager.cc:5495] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 1 to 3, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 3 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:12.770104 5272 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
W20260929 14:12:12.771733 5242 leader_election.cc:334] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.327s (SENT)
I20260929 14:12:12.776240 5274 raft_consensus.cc:1261] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260929 14:12:12.778278 7481 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260929 14:12:12.781471 5267 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
W20260929 14:12:12.789633 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 44621us
I20260929 14:12:12.798692 5242 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:12.750269 7494 consensus_queue.cc:1046] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20260929 14:12:12.797720 7432 consensus_queue.cc:260] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:12.842381 7432 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
W20260929 14:12:12.851151 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 85426.6us
W20260929 14:12:12.853430 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 134549us
I20260929 14:12:12.854724 5316 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:12.855294 6822 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:12.855767 6822 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:12.856120 6822 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:12.856583 5316 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:12.857430 7341 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:12.866034 7438 raft_consensus.cc:2726] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:12.865836 6822 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:12.865924 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 94202.5us
I20260929 14:12:12.887243 5873 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.891492 5096 catalog_manager.cc:5495] T 19257f7a5d364214b38d26c5f9b6d190 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 1 to 4, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 4 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:12.894187 5096 catalog_manager.cc:5495] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:12.878269 6868 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:12:12.890023 5098 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52736 (request call id 223) took 9628 ms. Trace:
I20260929 14:12:12.895985 6822 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:12.805779 7495 consensus_queue.cc:1035] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.071s
W20260929 14:12:12.898265 5242 leader_election.cc:334] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.670s (SENT)
I20260929 14:12:12.898742 5242 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.899644 7438 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260929 14:12:12.900066 5242 leader_election.cc:334] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 2.820s (SENT)
I20260929 14:12:12.896129 5098 rpcz_store.cc:276] 0929 14:12:03.261321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:03.261624 (+ 303us) service_pool.cc:227] Handling call
0929 14:12:03.694249 (+432625us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b54000f0d54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2af0 00007fbda4a75f2a 00007fbdaa9c7d6c 00007fbdaaac1c3e 00007fbda8dd3fb4 00007fbda8dd3f42 00007fbda8dd3e71 00007fbda8dd3dec 00007fbda8dd30a2 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:12.889791 (+9195542us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:12:05.052120 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:12:05.163713 (+111593us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:12:05.163747 (+ 34us) write_op.cc:435] Acquired schema lock
0929 14:12:05.163759 (+ 12us) tablet.cc:660] Decoding operations
0929 14:12:05.481197 (+317438us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:12:05.481282 (+ 85us) write_op.cc:641] Partition lock acquired for write op
0929 14:12:05.481306 (+ 24us) tablet.cc:683] Acquiring locks for 24 operations
0929 14:12:05.490655 (+ 9349us) tablet.cc:699] Row locks acquired
0929 14:12:05.490676 (+ 21us) write_op.cc:260] PREPARE: finished
0929 14:12:05.490829 (+ 153us) write_op.cc:270] Start()
0929 14:12:05.490997 (+ 168us) write_op.cc:276] Timestamp: P: 1790691125490776 usec, L: 0
0929 14:12:05.491029 (+ 32us) op_driver.cc:331] REPLICATION: starting
0929 14:12:05.587395 (+ 96366us) log.cc:842] Serialized 9426 byte log entry
0929 14:12:06.165547 (+578152us) op_driver.cc:444] REPLICATION: finished
0929 14:12:06.732591 (+567044us) write_op.cc:301] APPLY: starting
0929 14:12:06.735895 (+ 3304us) tablet.cc:1365] starting BulkCheckPresence
0929 14:12:06.851466 (+115571us) tablet.cc:1368] finished BulkCheckPresence
0929 14:12:06.969871 (+118405us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:12:08.040654 (+1070783us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:12:08.131887 (+ 91233us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=24
0929 14:12:08.131938 (+ 51us) write_op.cc:312] APPLY: finished
0929 14:12:08.132414 (+ 476us) log.cc:842] Serialized 164 byte log entry
0929 14:12:08.470811 (+338397us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:08.557776 (+ 86965us) write_op.cc:454] Released schema lock
0929 14:12:08.570665 (+ 12889us) write_op.cc:341] FINISH: Updating metrics
0929 14:12:08.592201 (+ 21536us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:08.592243 (+ 42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":8563,"reported_tablets":41,"spinlock_wait_cycles":490331904,"child_traces":[["op",{"apply.queue_time_us":505846,"apply.run_cpu_time_us":27331,"apply.run_wall_time_us":1867943,"num_ops":24,"prepare.queue_time_us":178881,"prepare.run_cpu_time_us":26885,"prepare.run_wall_time_us":751994,"raft.queue_time_us":219218,"raft.run_cpu_time_us":25592,"raft.run_wall_time_us":478086,"replication_time_us":656043,"thread_start_us":977068,"threads_started":5,"wal-append.queue_time_us":667322,"wal-append.run_cpu_time_us":11644,"wal-append.run_wall_time_us":2198694}]]}
I20260929 14:12:12.900491 5242 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.923380 5096 catalog_manager.cc:5495] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 2 to 4, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 4 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
W20260929 14:12:12.939441 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 83027us
I20260929 14:12:12.926014 7408 raft_consensus.cc:2726] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:12.938042 7495 raft_consensus.cc:2726] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260929 14:12:12.984000 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 83941.4us
W20260929 14:12:12.985081 5242 leader_election.cc:334] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 3.277s (SENT)
I20260929 14:12:12.985947 5242 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:12.994987 7379 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:12.996032 7406 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:13.002032 5242 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:13.009022 5195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 216) took 1093 ms. Trace:
I20260929 14:12:13.010255 7494 raft_consensus.cc:3032] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:13.010800 7494 raft_consensus.cc:729] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:13.014231 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 207) took 1212 ms. Trace:
I20260929 14:12:13.024993 7406 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:13.011842 7494 consensus_queue.cc:260] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:13.025571 7406 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:13.026273 7406 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:13.026530 5345 raft_consensus.cc:1261] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260929 14:12:13.009860 5195 rpcz_store.cc:276] 0929 14:12:11.915877 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:12.646236 (+730359us) service_pool.cc:227] Handling call
0929 14:12:12.666612 (+ 20376us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:12.667046 (+ 434us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:12.667078 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:12.667656 (+ 578us) log.cc:842] Serialized 26 byte log entry
0929 14:12:12.711662 (+ 44006us) raft_consensus.cc:1631] Marking committed up to 6
0929 14:12:12.925904 (+214242us) log.cc:842] Serialized 14 byte log entry
0929 14:12:13.008678 (+ 82774us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:13.008753 (+ 75us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:13.008792 (+ 39us) raft_consensus.cc:1671] finished
0929 14:12:13.008804 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:13.008937 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":82473,"prepare.queue_time_us":34254,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":1029,"raft.queue_time_us":102003,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":155,"thread_start_us":240976,"threads_started":2,"wal-append.queue_time_us":193594}
I20260929 14:12:13.028723 5242 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:13.028843 7412 consensus_queue.cc:1035] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260929 14:12:13.029906 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 66487.8us
I20260929 14:12:13.023356 5193 rpcz_store.cc:276] 0929 14:12:11.801514 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:12.563602 (+762088us) service_pool.cc:227] Handling call
0929 14:12:12.574533 (+ 10931us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:12.645912 (+ 71379us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:12.645951 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:12.649151 (+ 3200us) log.cc:842] Serialized 26 byte log entry
0929 14:12:12.791027 (+141876us) raft_consensus.cc:1631] Marking committed up to 7
0929 14:12:12.791387 (+ 360us) log.cc:842] Serialized 14 byte log entry
0929 14:12:12.977677 (+186290us) spinlock_profiling.cc:243] Waited 5.63 ms on lock 0x7b18003fa900. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1abc4ca 00007fbdab19df99 00007fbda1980990 00007fbda1b42864 00007fbda1b42155 00007fbda4a653d0 00007fbda4a77576 00007fbda4a880d7 00007fbda4a88058
0929 14:12:12.977935 (+ 258us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:12.978045 (+ 110us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:12.978105 (+ 60us) raft_consensus.cc:1671] finished
0929 14:12:13.013961 (+ 35856us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:13.014152 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42102,"prepare.queue_time_us":10033,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":51,"raft.queue_time_us":248234,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":5970,"spinlock_wait_cycles":12941184,"thread_start_us":117684,"threads_started":1,"wal-append.queue_time_us":212181}
I20260929 14:12:13.026536 7494 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:13.059551 7412 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:13.060243 7412 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:13.066649 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:13.077180 5197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1615) took 1288 ms. Trace:
I20260929 14:12:13.077881 5197 rpcz_store.cc:276] 0929 14:12:11.789063 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:12.564426 (+775363us) service_pool.cc:227] Handling call
0929 14:12:12.639874 (+ 75448us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:12.713311 (+ 73437us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:12.713344 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:12.729073 (+ 15729us) log.cc:842] Serialized 26 byte log entry
0929 14:12:12.923106 (+194033us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:12.923142 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:13.006149 (+ 83007us) spinlock_profiling.cc:243] Waited 51.8 ms on lock 0x7b54003db154. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:13.006206 (+ 57us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:13.069472 (+ 63266us) raft_consensus.cc:1671] finished
0929 14:12:13.069498 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:13.077094 (+ 7596us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34734,"raft.queue_time_us":15922,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":6891,"spinlock_wait_cycles":119084928,"thread_start_us":124671,"threads_started":1,"wal-append.queue_time_us":292062}
I20260929 14:12:13.057909 7408 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:13.086308 7408 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:13.086782 7408 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
W20260929 14:12:13.088867 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 60304.1us
I20260929 14:12:13.071992 7412 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:13.104441 5314 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:13.109341 5346 raft_consensus.cc:1226] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:13.119587 5242 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:13.126821 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 292152us
I20260929 14:12:13.131915 7040 raft_consensus.cc:2781] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:13.133038 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:13.142489 5314 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:13.122838 7432 consensus_queue.cc:1046] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.021s
I20260929 14:12:13.138993 7040 raft_consensus.cc:686] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:13.160748 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:13.172466 5314 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:13.168938 7357 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:13.177695 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:13.158785 7481 consensus_queue.cc:1035] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260929 14:12:13.191658 7357 raft_consensus.cc:686] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:13.211400 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:13.211382 7357 consensus_queue.cc:237] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:13.214655 7575 log.cc:924] Time spent T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
W20260929 14:12:13.217010 7574 log.cc:924] Time spent T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20260929 14:12:13.220677 7406 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:13.223448 7438 raft_consensus.cc:2726] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:13.226249 5344 raft_consensus.cc:1261] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:13.209834 7040 consensus_queue.cc:237] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:13.232187 5199 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 7.
I20260929 14:12:13.234444 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:13.191767 7311 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:13.223662 7406 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:13.203270 7379 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:13.244308 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:13.241518 7438 consensus_queue.cc:1035] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.020s
I20260929 14:12:13.259218 7311 raft_consensus.cc:686] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:13.296571 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:13.299002 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 49725.3us
I20260929 14:12:13.261804 7408 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:13.302963 5270 raft_consensus.cc:1261] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260929 14:12:13.313565 7412 raft_consensus.cc:3032] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:13.314042 7412 raft_consensus.cc:729] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:13.326566 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:13.274241 7481 consensus_queue.cc:1035] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
W20260929 14:12:13.333864 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 222254us
I20260929 14:12:13.335258 7412 consensus_queue.cc:260] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:13.336555 7412 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:13.325102 7311 consensus_queue.cc:237] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:13.351971 7572 log.cc:924] Time spent T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.163s user 0.000s sys 0.001s
I20260929 14:12:13.354118 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:13.371210 5276 raft_consensus.cc:1226] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:13.388638 5193 raft_consensus.cc:2396] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260929 14:12:13.385488 5242 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:13.390830 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:13.391438 5199 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
W20260929 14:12:13.394150 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 102306us
I20260929 14:12:13.394991 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:13.404829 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:13.396972 7438 raft_consensus.cc:2781] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260929 14:12:13.421296 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 56862.7us
I20260929 14:12:13.424134 5194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1622) took 1511 ms. Trace:
I20260929 14:12:13.424630 5194 rpcz_store.cc:276] 0929 14:12:11.910109 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:12.598561 (+688452us) service_pool.cc:227] Handling call
0929 14:12:12.598993 (+ 432us) raft_consensus.cc:1457] Updating replica for 2 ops
0929 14:12:12.644345 (+ 45352us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:12.644392 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0929 14:12:12.652890 (+ 8498us) log.cc:842] Serialized 52 byte log entry
0929 14:12:12.970215 (+317325us) raft_consensus.cc:1631] Marking committed up to 1
0929 14:12:12.970265 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:12.970351 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:13.421459 (+451108us) raft_consensus.cc:1671] finished
0929 14:12:13.421477 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:13.421600 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36821,"raft.queue_time_us":46958,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":5790,"thread_start_us":254959,"threads_started":1,"wal-append.queue_time_us":447840}
I20260929 14:12:13.416440 7438 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:13.435660 7438 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:13.435859 5242 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:13.435982 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:13.436770 5899 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:13.437263 5242 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:13.438683 5242 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:13.442222 7406 raft_consensus.cc:2726] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:13.442651 6868 consensus_queue.cc:1035] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260929 14:12:13.444339 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 175939us
I20260929 14:12:13.446522 5871 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:13.452157 7408 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:13.452410 7592 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260929 14:12:13.452754 7408 raft_consensus.cc:2726] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:13.451193 7438 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:13.464461 5906 raft_consensus.cc:2454] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:12:13.476140 7308 consensus_queue.cc:1035] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:12:13.483222 5267 raft_consensus.cc:1226] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:13.495433 5199 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:13.499562 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:13.535624 5899 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:13.473031 7406 raft_consensus.cc:2781] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260929 14:12:13.552572 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 77850.6us
I20260929 14:12:13.558403 7438 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:13.558815 7406 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:13.559242 7406 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:13.576145 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:13.586342 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 146518us
W20260929 14:12:13.590637 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 89777.9us
I20260929 14:12:13.599360 5873 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:13.600019 5347 raft_consensus.cc:1261] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260929 14:12:13.600651 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:12:13.607259 7597 log.cc:924] Time spent T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.064s user 0.000s sys 0.001s
I20260929 14:12:13.615084 5906 raft_consensus.cc:1261] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260929 14:12:13.615087 5271 raft_consensus.cc:1226] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:13.626183 7432 consensus_queue.cc:1035] T ae31abec82ff4f40b695ccb19f289dbc P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:13.629196 5199 raft_consensus.cc:1261] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260929 14:12:13.629945 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 191378us
I20260929 14:12:13.643069 7589 consensus_queue.cc:1035] T d653defbcc7e47ca89b35b1383e4e2c8 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:12:13.638526 7525 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.018s
I20260929 14:12:13.622535 7481 raft_consensus.cc:2781] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:13.673931 5242 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:13.676080 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 74503us
W20260929 14:12:13.720633 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 81609.1us
W20260929 14:12:13.722748 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 63637.5us
I20260929 14:12:13.737977 7598 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:13.732048 7589 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:13.737027 5242 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:13.734155 7601 raft_consensus.cc:2726] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:13.738567 7598 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:13.756137 7464 raft_consensus.cc:2781] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:13.761750 7464 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:13.762249 7464 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:13.748724 7406 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:13.826746 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 96649.1us
I20260929 14:12:13.753909 7589 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:13.854254 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 122604us
I20260929 14:12:13.764405 7357 consensus_queue.cc:1035] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.073s
I20260929 14:12:13.866407 7520 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.133s
W20260929 14:12:13.867152 7612 log.cc:924] Time spent T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29: Append to log took a long time: real 0.090s user 0.000s sys 0.002s
I20260929 14:12:13.855090 7540 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:13.925176 7580 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:13.926101 7580 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:13.928825 7580 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:13.924922 7613 log.cc:924] Time spent T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.165s user 0.000s sys 0.001s
I20260929 14:12:13.918269 7481 raft_consensus.cc:686] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [term 7 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
W20260929 14:12:13.944100 7614 log.cc:924] Time spent T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20260929 14:12:13.941365 7540 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:13.953472 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 232843us
I20260929 14:12:13.960172 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:13.969957 7406 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:13.976270 5165 leader_election.cc:334] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.927s (SENT)
I20260929 14:12:13.995126 5242 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:13.996217 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:12:14.017095 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 71185.5us
I20260929 14:12:14.042057 7598 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:14.048774 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:14.050107 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 87854.9us
I20260929 14:12:14.022235 7406 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:14.000748 7481 consensus_queue.cc:237] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:14.053426 7406 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:14.063539 7406 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:12:14.064445 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 60935us
I20260929 14:12:14.063553 7627 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:14.066668 7627 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:14.060698 7589 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:14.111158 7412 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:14.117347 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 95541.8us
I20260929 14:12:14.111757 7412 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:14.124115 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:14.124981 5199 raft_consensus.cc:2396] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e for term 9 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260929 14:12:14.121001 5242 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:14.123569 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 9 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.115272 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:14.132239 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:14.135139 7627 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:14.130540 5871 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:14.144172 5192 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 7.
I20260929 14:12:14.146989 5873 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:12:14.147959 7589 raft_consensus.cc:2726] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:14.149091 7589 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:14.149583 7589 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:14.149986 7589 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:14.151648 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:14.152285 5906 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 8.
I20260929 14:12:14.152369 7601 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:14.152971 7601 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:14.153461 7601 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:14.162454 5167 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:12:14.163291 7627 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:14.163676 7627 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:14.164014 7627 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:14.145421 7464 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:14.164999 7412 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:14.168546 5871 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:14.169360 7589 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:14.171228 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:14.172657 7589 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:14.182119 5316 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:14.183017 7600 log.cc:924] Time spent T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.509s user 0.007s sys 0.009s
I20260929 14:12:14.183794 7464 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:14.192308 5275 raft_consensus.cc:1261] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:14.193508 5271 raft_consensus.cc:1226] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:14.194746 5274 raft_consensus.cc:1226] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: []
W20260929 14:12:14.200824 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 94233.5us
I20260929 14:12:14.202972 5269 raft_consensus.cc:1261] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260929 14:12:14.203905 7525 raft_consensus.cc:2726] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:14.244923 5199 raft_consensus.cc:1261] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260929 14:12:14.255923 5192 raft_consensus.cc:1261] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260929 14:12:14.258122 5906 raft_consensus.cc:1226] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
W20260929 14:12:14.258487 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 111190us
I20260929 14:12:14.206898 7520 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:14.247072 7593 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:14.257417 5165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 2.200s (SENT) [suppressed 46 similar messages]
I20260929 14:12:14.256605 7476 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:12:14.314037 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 56622.4us
I20260929 14:12:14.257540 7616 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:14.331329 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 120092us
I20260929 14:12:14.286648 7412 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:14.352476 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 150784us
I20260929 14:12:14.328697 7321 consensus_queue.cc:1046] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260929 14:12:14.410547 7631 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:14.411437 7631 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:14.417518 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:14.433180 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:14.433200 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:14.436275 5192 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:14.438078 7406 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:14.440608 7406 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:14.441468 7406 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:14.444892 5192 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 8.
I20260929 14:12:14.442332 5899 raft_consensus.cc:1226] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:14.457917 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:14.458701 5200 raft_consensus.cc:1261] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:14.463452 7601 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:14.346365 7040 consensus_queue.cc:1046] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.118s
I20260929 14:12:14.473001 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:14.372184 7476 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:14.442368 7406 consensus_queue.cc:1225] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer ac17d3b553b74c9da39e0ab307addacd log is divergent from this leader: its last log entry 7.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260929 14:12:14.498526 6868 raft_consensus.cc:3032] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:14.499086 6868 raft_consensus.cc:729] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:14.500151 6868 consensus_queue.cc:260] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:14.393668 7616 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:14.527119 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.416301 7593 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:14.446733 7412 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:14.524804 5167 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:14.541363 7592 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:14.544229 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.519727 7379 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:14.480837 5316 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:14.569530 6868 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:14.562520 7379 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:14.575047 7379 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:14.577675 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:14.578792 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.579569 5267 raft_consensus.cc:2379] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 6: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:14.579284 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.580304 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.581863 5268 raft_consensus.cc:1226] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:14.581786 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:14.582566 5271 raft_consensus.cc:2354] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7.
I20260929 14:12:14.583050 5268 raft_consensus.cc:1226] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:14.583415 5271 raft_consensus.cc:1226] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:14.584117 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.580832 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.586164 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.601758 5192 raft_consensus.cc:1261] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260929 14:12:14.602530 7593 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:14.602857 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.603681 7601 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:14.603863 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:14.605757 5873 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:14.542128 7592 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:14.589365 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:14.614319 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.560477 7520 raft_consensus.cc:686] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:14.616662 5270 raft_consensus.cc:2354] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 6.
I20260929 14:12:14.616788 7357 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:14.619340 7520 consensus_queue.cc:237] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:14.617565 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 136862us
I20260929 14:12:14.620824 5316 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:14.587357 7643 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:14.620944 7626 consensus_queue.cc:1035] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260929 14:12:14.587256 7627 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:14.637989 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.615811 7379 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:14.639442 7379 raft_consensus.cc:686] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 8 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:14.640159 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.640463 7379 consensus_queue.cc:237] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:14.615571 7631 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:14.664719 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.721128 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:14.897230 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 372501us
I20260929 14:12:14.628743 7357 raft_consensus.cc:2726] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:14.798573 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 356270us
I20260929 14:12:14.898387 5316 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:14.629268 7643 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:14.721743 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.775377 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:14.776085 7540 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:14.698452 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.905622 5194 raft_consensus.cc:1226] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:14.907863 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:14.771804 7640 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:14.913697 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.795178 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.921236 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:14.935986 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.944105 5276 raft_consensus.cc:2354] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7.
I20260929 14:12:14.945272 5345 raft_consensus.cc:2379] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:14.941746 5195 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:12:14.908928 7269 raft_consensus.cc:3032] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:14.765338 7464 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:14.704577 7476 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:14.665824 7646 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:14.920435 7631 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:14.971318 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.971318 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:14.796080 7627 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:14.967907 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:14.974181 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 75862.7us
W20260929 14:12:14.974823 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 71718.5us
I20260929 14:12:14.974975 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:14.945158 7540 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:14.942910 7616 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:14.951313 7357 raft_consensus.cc:2781] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 14:12:15.039866 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:15.050294 5194 raft_consensus.cc:2454] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:12:15.041440 7406 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:15.052095 7649 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:15.052250 7406 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.054339 7649 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:15.054666 5165 leader_election.cc:334] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 3.084s (SENT)
I20260929 14:12:15.054837 7649 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.055253 5165 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:15.042605 7540 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.041626 7646 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.034171 7601 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:15.049407 5346 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:12:15.060433 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.102607 5270 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:15.103338 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:15.103747 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.104907 5270 raft_consensus.cc:2396] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260929 14:12:15.061669 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.105785 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.106427 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.107136 5269 raft_consensus.cc:2379] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 5: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:15.061969 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.108229 5275 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
W20260929 14:12:15.108527 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 130370us
I20260929 14:12:15.062505 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.109752 5271 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:12:15.065580 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.068637 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.070366 5276 raft_consensus.cc:1226] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:15.112064 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:15.061098 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.113883 5269 raft_consensus.cc:2354] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7.
I20260929 14:12:14.962116 7269 raft_consensus.cc:729] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:15.032279 7592 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:15.106454 5270 raft_consensus.cc:2379] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 7: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:15.062472 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.032415 7476 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:15.507450 5276 raft_consensus.cc:1226] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:15.507328 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.509537 5275 raft_consensus.cc:2379] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 5: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:15.180471 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:15.205418 5165 leader_election.cc:334] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.537s (SENT)
I20260929 14:12:15.512974 5165 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:15.213428 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.214087 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:15.508491 5268 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:12:15.432932 5873 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:15.212541 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:15.127888 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 52165.5us
W20260929 14:12:15.205775 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 159284us
I20260929 14:12:15.517756 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:15.517534 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:15.080606 7644 log.cc:924] Time spent T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.157s user 0.000s sys 0.001s
I20260929 14:12:15.217746 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:15.528679 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:15.528950 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 96150us
I20260929 14:12:15.532385 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.532516 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.063897 7406 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.532830 5873 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:12:15.093175 7616 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:15.536594 5268 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:15.530336 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.539376 5275 raft_consensus.cc:2379] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 5: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:15.065032 7540 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.076797 7643 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.067677 7646 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.495045 7311 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:15.041929 7464 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:15.543005 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 433775us
I20260929 14:12:15.215600 6868 consensus_queue.cc:1046] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.339s
I20260929 14:12:15.530210 7654 raft_consensus.cc:2726] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:15.065943 7656 raft_consensus.cc:2726] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:12:15.078691 7412 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:15.216187 7637 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:15.547972 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.548561 5196 raft_consensus.cc:1261] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260929 14:12:15.581557 5196 pending_rounds.cc:76] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd: Aborting all ops after (but not including) 1
I20260929 14:12:15.581799 5196 pending_rounds.cc:97] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260929 14:12:15.582034 5196 raft_consensus.cc:2866] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:12:15.549710 7675 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:15.217070 5343 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:15.592540 5343 raft_consensus.cc:595] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:15.592934 5343 raft_consensus.cc:603] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:12:15.057200 7649 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.542049 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.217299 7357 raft_consensus.cc:686] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:15.594177 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.594431 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:15.594921 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 406586us
I20260929 14:12:15.594476 7357 consensus_queue.cc:237] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:15.555244 7616 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:15.599865 7616 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.218379 7269 consensus_queue.cc:260] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.601883 7269 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:15.613866 5197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1651) took 1405 ms. Trace:
I20260929 14:12:15.614310 5197 rpcz_store.cc:276] 0929 14:12:14.208700 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:14.208870 (+ 170us) service_pool.cc:227] Handling call
0929 14:12:14.209292 (+ 422us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:14.246604 (+ 37312us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:14.246650 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:14.450479 (+203829us) log.cc:842] Serialized 26 byte log entry
0929 14:12:14.650739 (+200260us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:14.961716 (+310977us) log.cc:842] Serialized 14 byte log entry
0929 14:12:14.964435 (+ 2719us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:14.964506 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:15.610415 (+645909us) spinlock_profiling.cc:243] Waited 37.9 ms on lock 0x7b1800048160. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b45638 00007fbda1b48b11 00007fbda1b48ac9 00007fbda1b48a59 00007fbda1b48a21 00007fbda1b47d1d 00007fbdaada9204 00007fbdaada9039 00007fbda1b27dc6
0929 14:12:15.613170 (+ 2755us) raft_consensus.cc:1671] finished
0929 14:12:15.613661 (+ 491us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:15.613780 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7135,"prepare.queue_time_us":380795,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":7608,"raft.queue_time_us":149193,"raft.run_cpu_time_us":320,"raft.run_wall_time_us":61505,"spinlock_wait_cycles":87117312,"thread_start_us":435763,"threads_started":2,"wal-append.queue_time_us":593616}
I20260929 14:12:15.574397 7311 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:15.546583 7593 consensus_queue.cc:1046] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260929 14:12:15.617726 7341 raft_consensus.cc:3032] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:15.618103 7341 raft_consensus.cc:729] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:15.618160 7637 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.618975 7341 consensus_queue.cc:260] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.619493 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:15.620201 7341 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:15.533692 7592 raft_consensus.cc:2781] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:15.620813 7592 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:15.621168 7592 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:15.547387 7476 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.623317 7476 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.562667 7321 consensus_queue.cc:1046] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.062s
I20260929 14:12:15.562402 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.625706 5268 raft_consensus.cc:2379] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 5: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:15.628546 7464 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:15.629027 7464 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:15.629462 7464 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.631273 7464 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.555102 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.632534 5276 raft_consensus.cc:2379] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 6: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:15.633714 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:15.563376 5271 raft_consensus.cc:1226] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:15.635803 5273 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:12:15.065337 7520 consensus_queue.cc:1046] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.483s
I20260929 14:12:15.066056 7652 consensus_queue.cc:1046] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.583s
I20260929 14:12:15.548180 7406 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:15.567765 7302 raft_consensus.cc:2781] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260929 14:12:15.573920 7664 log.cc:924] Time spent T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260929 14:12:15.075613 7601 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.092334 7631 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.189852 7041 consensus_queue.cc:1035] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.366s
I20260929 14:12:14.947467 7640 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.643025 7641 raft_consensus.cc:3032] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:15.641291 7406 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.643522 7641 raft_consensus.cc:729] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:15.643985 7640 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.644160 7601 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.644528 7640 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:15.644693 7601 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:15.585834 5270 raft_consensus.cc:2396] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 6 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260929 14:12:15.581043 5274 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:12:15.620237 7637 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.545922 5269 raft_consensus.cc:1226] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:15.641115 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.625078 7321 raft_consensus.cc:3032] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:12:15.645367 5267 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:15.647294 5272 raft_consensus.cc:2396] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260929 14:12:15.641238 7379 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.576s user 0.005s sys 0.105s
I20260929 14:12:15.648581 7520 raft_consensus.cc:3032] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:15.647205 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.648335 5275 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:12:15.652758 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.654843 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.645032 7406 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.640379 7302 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:15.657119 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.657123 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.659359 5344 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:15.660765 5275 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
W20260929 14:12:15.652091 7379 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s user 0.005s sys 0.105s
I20260929 14:12:15.662281 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.662142 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.666641 5271 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:12:15.667008 5270 raft_consensus.cc:2354] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:12:15.667943 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.660766 5276 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:15.661172 5344 raft_consensus.cc:595] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:15.665617 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:15.644994 7640 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.671126 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.672188 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.672716 7640 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.672915 5271 raft_consensus.cc:2396] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260929 14:12:15.644389 7641 consensus_queue.cc:260] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:15.674556 7641 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:15.675675 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.676479 5269 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:15.649734 7321 raft_consensus.cc:729] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:15.685604 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.686638 5273 raft_consensus.cc:2379] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 5: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:15.690045 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1656) took 1408 ms. Trace:
I20260929 14:12:15.690091 7406 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:15.653818 7520 raft_consensus.cc:729] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:15.690819 7406 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.691780 7520 consensus_queue.cc:260] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.692615 7674 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:15.647863 7675 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.693182 7674 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.662699 7302 raft_consensus.cc:3037] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:15.690637 5193 rpcz_store.cc:276] 0929 14:12:14.281297 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:14.294155 (+ 12858us) service_pool.cc:227] Handling call
0929 14:12:14.640624 (+346469us) spinlock_profiling.cc:243] Waited 14.2 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:12:14.640808 (+ 184us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:14.902853 (+262045us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:14.902882 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:14.903755 (+ 873us) log.cc:842] Serialized 26 byte log entry
0929 14:12:15.059357 (+155602us) spinlock_profiling.cc:243] Waited 974 us on lock 0x7b1800387580. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda46fc82c 00007fbda1b279dc 00007fbdaa95ea39 00007fbda1b448bb 00007fbda1b42cbf 00007fbda1b44e22 00007fbda46ee5bb 00007fbda46f4063 00007fbda46f4411
0929 14:12:15.059552 (+ 195us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:15.059902 (+ 350us) log.cc:842] Serialized 14 byte log entry
0929 14:12:15.060098 (+ 196us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:15.060168 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:15.689850 (+629682us) raft_consensus.cc:1671] finished
0929 14:12:15.689870 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:15.689988 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4749,"prepare.queue_time_us":9659,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":322627,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":23689,"spinlock_wait_cycles":34965504,"thread_start_us":328551,"threads_started":2,"wal-append.queue_time_us":335278}
I20260929 14:12:15.645947 7601 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.700640 7637 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:15.701251 7637 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.722707 5268 raft_consensus.cc:2354] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7.
I20260929 14:12:15.727310 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.731191 7676 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:15.731786 7676 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.735234 5276 raft_consensus.cc:2354] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7.
I20260929 14:12:15.725378 7640 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:15.762224 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.763171 5275 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:12:15.684397 7321 consensus_queue.cc:260] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.789927 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.790540 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.790879 5276 raft_consensus.cc:2396] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260929 14:12:15.803155 5268 raft_consensus.cc:2379] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 7: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:15.803606 5344 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:15.805969 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.806826 5275 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
W20260929 14:12:15.807760 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 252638us
W20260929 14:12:15.808956 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 276168us
I20260929 14:12:15.803526 5098 catalog_manager.cc:5495] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
W20260929 14:12:15.808410 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 255001us
I20260929 14:12:15.809183 5870 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:15.804334 5344 raft_consensus.cc:595] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:15.814682 5098 catalog_manager.cc:5495] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 6 to 8, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 8 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260929 14:12:15.822463 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:15.823078 5906 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:15.822901 7430 consensus_queue.cc:1035] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:12:15.823819 5098 catalog_manager.cc:5495] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:15.798302 7311 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:15.802515 7520 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:15.825872 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:12:15.838130 5096 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52726 (request call id 226) took 11873 ms (client timeout 15000 ms). Trace:
I20260929 14:12:15.780097 7640 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:15.842979 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.843559 5276 raft_consensus.cc:2354] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7.
I20260929 14:12:15.843699 5270 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
W20260929 14:12:15.844858 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 237391us
W20260929 14:12:15.845957 5165 leader_election.cc:334] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.591s (SENT)
I20260929 14:12:15.846846 5906 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 9.
I20260929 14:12:15.848799 5268 raft_consensus.cc:1261] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260929 14:12:15.846882 5165 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:15.848946 5098 catalog_manager.cc:5495] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 3 to 7, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 7 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:15.855500 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.856511 5344 raft_consensus.cc:3037] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:15.847855 7406 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.861882 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:15.862722 5267 raft_consensus.cc:1226] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:15.863027 7321 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:15.863152 7676 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.866640 5345 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
W20260929 14:12:15.868788 5165 leader_election.cc:334] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.600s (SENT)
W20260929 14:12:15.847891 5096 rpcz_store.cc:269] 0929 14:12:03.964585 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:03.964832 (+ 247us) service_pool.cc:227] Handling call
0929 14:12:15.838060 (+11873228us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:12:13.585193 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:12:13.596444 (+ 11251us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:12:13.596497 (+ 53us) write_op.cc:435] Acquired schema lock
0929 14:12:13.596525 (+ 28us) tablet.cc:660] Decoding operations
0929 14:12:13.597726 (+ 1201us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:12:13.597819 (+ 93us) write_op.cc:641] Partition lock acquired for write op
0929 14:12:13.597854 (+ 35us) tablet.cc:683] Acquiring locks for 10 operations
0929 14:12:13.598863 (+ 1009us) tablet.cc:699] Row locks acquired
0929 14:12:13.598894 (+ 31us) write_op.cc:260] PREPARE: finished
0929 14:12:13.599073 (+ 179us) write_op.cc:270] Start()
0929 14:12:13.599216 (+ 143us) write_op.cc:276] Timestamp: P: 1790691133599006 usec, L: 0
0929 14:12:13.599284 (+ 68us) op_driver.cc:331] REPLICATION: starting
0929 14:12:13.618523 (+ 19239us) log.cc:842] Serialized 3974 byte log entry
0929 14:12:14.058951 (+440428us) op_driver.cc:444] REPLICATION: finished
0929 14:12:14.381020 (+322069us) write_op.cc:301] APPLY: starting
0929 14:12:14.444968 (+ 63948us) tablet.cc:1365] starting BulkCheckPresence
0929 14:12:14.465199 (+ 20231us) tablet.cc:1368] finished BulkCheckPresence
0929 14:12:14.502054 (+ 36855us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:12:15.073384 (+571330us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:12:15.073753 (+ 369us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0929 14:12:15.073805 (+ 52us) write_op.cc:312] APPLY: finished
0929 14:12:15.074237 (+ 432us) log.cc:842] Serialized 77 byte log entry
0929 14:12:15.547029 (+472792us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:15.558344 (+ 11315us) write_op.cc:454] Released schema lock
0929 14:12:15.579275 (+ 20931us) write_op.cc:341] FINISH: Updating metrics
0929 14:12:15.593472 (+ 14197us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:15.593504 (+ 32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":199,"reported_tablets":40,"child_traces":[["op",{"apply.queue_time_us":221451,"apply.run_cpu_time_us":20523,"apply.run_wall_time_us":1271812,"num_ops":10,"prepare.queue_time_us":294192,"prepare.run_cpu_time_us":43273,"prepare.run_wall_time_us":150435,"raft.queue_time_us":71748,"raft.run_cpu_time_us":52435,"raft.run_wall_time_us":149285,"replication_time_us":458527,"thread_start_us":774015,"threads_started":5,"wal-append.queue_time_us":572791,"wal-append.run_cpu_time_us":16097,"wal-append.run_wall_time_us":1295529}]]}
W20260929 14:12:15.869939 5165 leader_election.cc:334] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.695s (SENT)
I20260929 14:12:15.870365 5165 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:15.873798 5274 raft_consensus.cc:1261] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260929 14:12:15.874183 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 316991us
I20260929 14:12:15.876266 5275 raft_consensus.cc:2396] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260929 14:12:15.867702 5345 raft_consensus.cc:595] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:15.841151 7379 raft_consensus.cc:2781] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:15.875581 5098 catalog_manager.cc:5495] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:15.882267 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:15.889878 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.890573 5273 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:15.884407 5098 catalog_manager.cc:5495] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 3 to 5, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 5 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:15.893477 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.857256 7601 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.894343 5272 raft_consensus.cc:2396] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260929 14:12:15.892120 5098 catalog_manager.cc:5495] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 2 to 4, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:15.871034 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:15.901247 5167 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:12:15.902472 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:15.908373 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:15.906240 5098 catalog_manager.cc:5495] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:15.895758 7656 raft_consensus.cc:2726] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:15.895206 7649 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:15.914889 7412 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election won for term 9
I20260929 14:12:15.881744 7674 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.883551 7379 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:15.895789 7476 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:15.902374 7631 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.966698 7476 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:15.968786 7476 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:15.970830 7651 raft_consensus.cc:3032] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Stepping down as leader of term 7
I20260929 14:12:15.971406 7651 raft_consensus.cc:729] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:15.972100 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:15.972242 7651 consensus_queue.cc:260] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:15.918833 7622 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:15.973559 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 103643us
I20260929 14:12:15.973718 7476 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:15.974681 7476 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:15.976668 5165 leader_election.cc:334] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.761s (SENT)
I20260929 14:12:15.977394 7656 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:15.977509 5165 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:15.978030 7656 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:15.978785 7631 raft_consensus.cc:2726] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:15.982623 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 64540.6us
I20260929 14:12:15.920902 5098 catalog_manager.cc:5495] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 3 to 4, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:15.996718 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:15.997476 5270 raft_consensus.cc:2354] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:12:15.889179 7675 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.980415 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:15.893206 7643 raft_consensus.cc:2726] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:15.997557 5098 catalog_manager.cc:5495] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 5 to 6, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 6 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:16.005199 5098 catalog_manager.cc:5495] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 4 to 5, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 5 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:16.006660 5351 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:16.007246 5351 raft_consensus.cc:595] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:16.008404 5351 raft_consensus.cc:603] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:12:15.899353 7676 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:16.007419 5098 catalog_manager.cc:5495] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 3 to 4, leader changed from 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 4 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:15.898114 7687 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:15.899317 7637 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:16.014770 5098 catalog_manager.cc:5495] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:16.026515 5098 catalog_manager.cc:5495] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:16.029017 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.029937 5269 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:16.025439 7688 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:16.005031 7622 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:16.032015 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 55402.1us
I20260929 14:12:16.031880 7688 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:16.033421 7622 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:16.025596 7676 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:16.034806 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:16.035424 5196 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 8.
W20260929 14:12:16.035612 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 497501us
I20260929 14:12:16.011685 7412 raft_consensus.cc:686] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:16.036369 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.037039 5275 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:16.035754 7676 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:15.979663 7379 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:16.028410 5098 catalog_manager.cc:5495] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 1 to 6, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 6 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:16.055488 7640 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:16.001421 7649 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:16.043423 7687 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:16.057509 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.058195 7412 consensus_queue.cc:237] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:16.058723 7656 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:16.059490 7688 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:16.060034 7592 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:16.061549 5899 raft_consensus.cc:2379] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:16.061064 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:16.106655 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 55162.3us
I20260929 14:12:16.069169 5873 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:16.011031 7651 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:16.079131 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.078188 5098 catalog_manager.cc:5495] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:16.078061 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.094740 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.111276 5275 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:12:16.095407 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:16.099156 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.111749 5098 catalog_manager.cc:5495] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:16.102727 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.100083 5274 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:16.104697 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.115164 5268 raft_consensus.cc:2354] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:12:16.104784 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:16.113272 5098 catalog_manager.cc:5495] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 4 to 5, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 5 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:16.115705 5193 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:12:16.112360 5194 raft_consensus.cc:2454] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 2.
I20260929 14:12:16.123620 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.098723 7637 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:16.125975 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 56934.3us
I20260929 14:12:16.129522 5314 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:16.134027 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.136369 5275 raft_consensus.cc:1261] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:16.137630 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.143134 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260929 14:12:16.149902 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.153100 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:16.150828 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:12:16.141331 7648 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:16.116945 7525 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:16.145814 7430 consensus_queue.cc:1035] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20260929 14:12:16.158965 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.166246 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:16.167332 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:16.168105 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:16.153965 5197 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:12:16.169265 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.170334 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.166651 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:16.171627 5165 leader_election.cc:334] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.768s (SENT)
I20260929 14:12:16.172065 5165 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:16.172379 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:16.173667 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:16.175298 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.178277 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:16.181633 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:16.166102 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:16.183445 7686 consensus_queue.cc:1035] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
I20260929 14:12:16.184137 5901 raft_consensus.cc:2379] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 6: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:12:16.157831 7357 raft_consensus.cc:2781] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:16.184928 7357 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:16.185055 7617 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:16.185361 7357 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:16.185591 7617 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:16.186050 7617 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:16.189764 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.166693 7648 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:16.166992 7525 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:16.183338 7658 consensus_queue.cc:1035] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.084s
I20260929 14:12:16.195497 7430 raft_consensus.cc:979] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:16.195945 7430 raft_consensus.cc:1067] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
W20260929 14:12:16.196882 7302 consensus_meta.cc:287] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.501s user 0.005s sys 0.005s
I20260929 14:12:16.197506 7592 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:16.198432 5192 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:16.183297 5197 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:16.189853 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.200976 5273 raft_consensus.cc:2379] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 4: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:16.201124 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.203305 5344 raft_consensus.cc:1261] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260929 14:12:16.193608 7525 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:16.204952 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.206110 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.207329 7041 consensus_queue.cc:1035] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260929 14:12:16.214299 7602 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:16.214802 7602 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:16.135433 7637 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:16.207038 7656 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:16.174777 7476 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:16.227877 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 47431us
W20260929 14:12:16.235037 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 63430.5us
I20260929 14:12:16.238368 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:16.238401 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 59194.1us
W20260929 14:12:16.241694 5165 leader_election.cc:334] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.769s (SENT)
I20260929 14:12:16.241645 5873 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:12:16.242395 5165 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:16.244343 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.246639 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.247762 5272 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:12:16.237161 6868 consensus_queue.cc:1046] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20260929 14:12:16.241386 5242 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:16.254024 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:16.257977 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.255885 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.205325 7302 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:16.247653 7656 raft_consensus.cc:2726] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260929 14:12:16.264559 5165 leader_election.cc:334] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.811s (SENT)
I20260929 14:12:16.265169 5165 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:16.266450 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.274215 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.237531 7676 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.026s
I20260929 14:12:16.286814 5906 raft_consensus.cc:1226] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:16.259940 7640 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:16.291401 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:16.293038 7341 raft_consensus.cc:979] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:16.297716 7341 raft_consensus.cc:1067] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:16.299288 7647 raft_consensus.cc:979] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
W20260929 14:12:16.300312 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 54531.2us
I20260929 14:12:16.300489 7647 raft_consensus.cc:1067] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:16.301482 7430 consensus_queue.cc:1035] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20260929 14:12:16.301934 7692 log.cc:924] Time spent T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.093s user 0.000s sys 0.000s
I20260929 14:12:16.303975 7302 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:16.278635 7648 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:16.305168 7648 consensus_queue.cc:1035] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20260929 14:12:16.305838 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 53606.2us
I20260929 14:12:16.306257 7652 consensus_queue.cc:1035] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260929 14:12:16.310534 5197 raft_consensus.cc:1261] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260929 14:12:16.310925 5197 pending_rounds.cc:76] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd: Aborting all ops after (but not including) 4
I20260929 14:12:16.311172 5197 pending_rounds.cc:97] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260929 14:12:16.311403 5197 raft_consensus.cc:2866] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:12:16.312748 5192 raft_consensus.cc:1261] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260929 14:12:16.310209 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.313701 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:16.314311 7637 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:16.323566 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:16.323876 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.324888 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:16.274013 7476 raft_consensus.cc:2726] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:16.338434 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.345811 5200 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:12:16.346401 5200 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:16.346760 5200 raft_consensus.cc:3037] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:16.278443 7687 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:16.290004 7649 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:16.280668 7673 raft_consensus.cc:2781] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:16.352846 7673 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:16.353325 7673 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:16.290540 7640 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:16.355134 7640 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:16.356707 7691 raft_consensus.cc:3032] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:16.357170 7691 raft_consensus.cc:729] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:16.358198 7691 consensus_queue.cc:260] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:16.359715 7691 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:16.364524 5196 raft_consensus.cc:1261] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260929 14:12:16.364909 5196 pending_rounds.cc:76] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd: Aborting all ops after (but not including) 3
I20260929 14:12:16.365103 5196 pending_rounds.cc:97] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260929 14:12:16.365340 5196 raft_consensus.cc:2866] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:12:16.373999 7676 consensus_queue.cc:1046] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260929 14:12:16.374996 7676 raft_consensus.cc:3032] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:12:16.375325 7676 raft_consensus.cc:729] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:16.378906 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.379861 5899 raft_consensus.cc:1226] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9. Ops: [4.6-8.8]
W20260929 14:12:16.382489 7641 consensus_meta.cc:287] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.707s user 0.009s sys 0.005s
I20260929 14:12:16.380975 5906 raft_consensus.cc:1261] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260929 14:12:16.382010 7602 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:16.383195 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 122001us
W20260929 14:12:16.384759 5165 leader_election.cc:334] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.819s (SENT)
I20260929 14:12:16.385308 5165 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:16.389503 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 128280us
W20260929 14:12:16.400278 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 56728.5us
I20260929 14:12:16.390499 5167 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:16.395627 5200 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:16.411000 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:16.413367 7412 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:16.414563 5195 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
W20260929 14:12:16.418319 5165 leader_election.cc:334] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.836s (SENT)
I20260929 14:12:16.418751 5165 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:16.419008 5197 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:12:16.419835 5197 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:16.417676 5167 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:16.420195 5197 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:16.413337 7602 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260929 14:12:16.421514 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:16.422467 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:16.390933 7649 raft_consensus.cc:2726] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:16.423065 5316 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:16.420337 5200 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:16.425767 5346 raft_consensus.cc:3032] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:16.429499 5346 raft_consensus.cc:729] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
W20260929 14:12:16.433162 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 129878us
I20260929 14:12:16.437528 5195 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:16.438963 5195 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:16.434636 7308 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:16.441627 5272 raft_consensus.cc:1226] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:16.436498 5167 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:16.446625 7321 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 14:12:16.390882 7637 consensus_queue.cc:1046] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260929 14:12:16.422142 7476 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:16.417635 7676 consensus_queue.cc:260] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:16.457875 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:16.458674 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:16.386528 7379 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:16.463924 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.448202 5346 consensus_queue.cc:260] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:16.431530 7643 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:16.471702 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:16.477269 5346 raft_consensus.cc:3037] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:12:16.477854 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 59568.2us
W20260929 14:12:16.478585 5165 leader_election.cc:334] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.861s (SENT)
I20260929 14:12:16.479065 5165 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:16.494292 7680 consensus_queue.cc:1035] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.080s
I20260929 14:12:16.447813 7649 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:16.511493 5346 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:16.454352 7690 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:16.521153 5197 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:16.496984 7677 raft_consensus.cc:3032] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Stepping down as leader of term 8
W20260929 14:12:16.535729 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 57179.1us
W20260929 14:12:16.536633 5165 leader_election.cc:334] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.888s (SENT)
I20260929 14:12:16.537108 5165 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:16.539325 5351 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 9. (term mismatch)
I20260929 14:12:16.539801 5351 pending_rounds.cc:76] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4: Aborting all ops after (but not including) 8
I20260929 14:12:16.540066 5351 pending_rounds.cc:97] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4: Aborting uncommitted NO_OP operation due to leader change: 7.9
I20260929 14:12:16.444983 7617 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:16.541728 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.544653 5164 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:16.548123 5314 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:16.508589 7540 raft_consensus.cc:2726] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:16.513110 7430 consensus_queue.cc:1046] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
W20260929 14:12:16.551823 7706 log.cc:924] Time spent T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260929 14:12:16.552197 5898 raft_consensus.cc:2379] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 5: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
W20260929 14:12:16.553793 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 63772us
I20260929 14:12:16.540374 5351 raft_consensus.cc:2866] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:12:16.565176 5197 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:16.568172 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:16.568819 5346 raft_consensus.cc:3032] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:12:16.569142 5346 raft_consensus.cc:729] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:16.538187 7676 raft_consensus.cc:3037] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:16.532696 7677 raft_consensus.cc:729] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:16.575438 5346 consensus_queue.cc:260] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:16.576388 5346 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:16.584569 7412 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:16.564288 7649 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:16.560936 7308 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260929 14:12:16.591262 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 54677.9us
I20260929 14:12:16.592285 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.593014 5351 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 6.
I20260929 14:12:16.594278 7640 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:16.596824 7702 consensus_queue.cc:1035] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260929 14:12:16.597412 7640 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:16.588378 5346 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
W20260929 14:12:16.598552 5165 leader_election.cc:334] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.905s (SENT)
I20260929 14:12:16.599088 5165 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:16.600240 5165 leader_election.cc:334] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.905s (SENT)
I20260929 14:12:16.600641 5165 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:16.602034 5164 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:16.553669 7690 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:16.608217 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.610409 7617 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 9 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:16.610905 5316 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:16.622332 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.582459 7540 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:16.582803 7643 raft_consensus.cc:2781] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:16.622305 5870 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:16.628870 7688 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260929 14:12:16.617959 7357 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:16.630579 7690 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:16.631032 5350 raft_consensus.cc:2379] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 5: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:16.631135 7357 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:16.631529 5165 leader_election.cc:334] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.916s (SENT)
I20260929 14:12:16.632145 5165 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:16.632930 7412 raft_consensus.cc:2726] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:16.633203 5165 leader_election.cc:334] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.956s (SENT)
I20260929 14:12:16.633669 5165 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:16.634011 7643 raft_consensus.cc:686] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:16.634637 7678 raft_consensus.cc:2781] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:16.626600 7649 raft_consensus.cc:2726] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:16.635658 7649 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:16.636456 7631 raft_consensus.cc:2781] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:12:16.637080 7631 raft_consensus.cc:686] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:16.625108 7652 raft_consensus.cc:2726] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:16.638486 7540 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:16.635149 7678 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:16.640054 7678 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:16.640683 7540 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:16.635124 7643 consensus_queue.cc:237] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:16.633761 7379 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:16.644603 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:16.651660 5267 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:16.666204 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:16.668437 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:16.669062 5906 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 8.
I20260929 14:12:16.674975 5167 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:12:16.625874 7617 consensus_queue.cc:1035] T 4040985909414481ad0d1491249e8e67 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260929 14:12:16.681723 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:16.684955 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:12:16.666234 7711 log.cc:924] Time spent T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20260929 14:12:16.706584 5906 raft_consensus.cc:1226] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:16.712381 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:16.705032 5267 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:12:16.612217 7677 consensus_queue.cc:260] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:16.728070 5870 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:16.714993 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:16.727670 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:16.743907 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 56646.7us
I20260929 14:12:16.744617 5871 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:16.747413 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:12:16.755194 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 65214.1us
I20260929 14:12:16.638530 7631 consensus_queue.cc:237] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:16.749379 7598 raft_consensus.cc:2726] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:16.780663 7629 raft_consensus.cc:2781] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:16.749331 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:16.743536 7640 consensus_queue.cc:1046] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
W20260929 14:12:16.804735 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 128374us
I20260929 14:12:16.776589 7677 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Advancing to term 9
W20260929 14:12:16.806699 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 173500us
I20260929 14:12:16.827327 5906 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260929 14:12:16.834227 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:16.834425 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:16.832429 5195 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:16.885547 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 56508.5us
W20260929 14:12:16.888784 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 73875.3us
I20260929 14:12:16.806521 7629 raft_consensus.cc:686] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 5 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:16.888365 5192 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:16.947131 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:16.947811 5201 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:16.829201 7649 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260929 14:12:16.887948 7670 log.cc:924] Time spent T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
W20260929 14:12:16.978603 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 70338.5us
I20260929 14:12:16.979748 5239 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:16.978191 7648 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:17.006747 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:17.007735 5192 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
W20260929 14:12:16.999265 7673 consensus_meta.cc:287] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.645s user 0.004s sys 0.007s
I20260929 14:12:16.955509 7629 consensus_queue.cc:237] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:17.011955 5193 raft_consensus.cc:1261] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:16.896144 7656 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.110s
I20260929 14:12:16.993964 7640 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:17.011497 7476 consensus_meta.cc:287] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.531s user 0.009s sys 0.001s
W20260929 14:12:17.041690 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 205878us
W20260929 14:12:17.044196 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 54157us
I20260929 14:12:17.003791 7648 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:17.049484 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:17.050151 5193 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:17.036809 7476 raft_consensus.cc:2726] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:17.055441 5192 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:17.060454 7649 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:17.061157 7649 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:17.062446 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:17.063061 5196 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:17.066537 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:17.067122 5192 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:17.080232 5192 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:17.081550 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:17.085623 5165 leader_election.cc:334] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.973s (SENT)
I20260929 14:12:17.086133 5165 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:17.086823 5242 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:17.075346 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 73892.8us
I20260929 14:12:17.088832 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:17.093004 7648 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:17.089727 5192 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:17.095386 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:17.096642 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:17.098500 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 52053.5us
W20260929 14:12:17.102092 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 51520.7us
I20260929 14:12:17.103284 5906 raft_consensus.cc:1226] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:17.100659 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:17.097461 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:17.104542 5192 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:17.105788 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:17.107321 5192 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:17.108584 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:17.110662 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:17.027623 7673 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:17.112301 5193 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:17.118805 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:17.120844 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:17.121616 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:12:17.124502 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 53350.9us
I20260929 14:12:17.124687 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:17.126731 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:17.127023 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:17.127156 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:17.127727 5192 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 7.
I20260929 14:12:17.125715 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:17.127866 5271 raft_consensus.cc:3037] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:17.128583 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:17.130240 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:17.118595 7540 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:17.131438 5242 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:17.118161 7676 raft_consensus.cc:2726] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:17.138414 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:17.139914 5270 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:17.143585 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:17.143836 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 70367.5us
W20260929 14:12:17.473978 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 347148us
I20260929 14:12:17.298210 5196 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
W20260929 14:12:17.475711 5165 leader_election.cc:334] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.976s (SENT)
I20260929 14:12:17.478657 5276 raft_consensus.cc:2454] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:12:17.476747 5242 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:17.479022 5165 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:17.361457 6868 raft_consensus.cc:3032] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
W20260929 14:12:17.483772 5165 leader_election.cc:334] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.984s (SENT)
I20260929 14:12:17.288288 5201 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:17.484277 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:17.494436 5165 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:17.495797 5198 raft_consensus.cc:1226] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:17.484308 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:17.485450 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 367864us
I20260929 14:12:17.485903 7430 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:17.281680 7728 log.cc:924] Time spent T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.162s user 0.000s sys 0.001s
W20260929 14:12:17.469259 7677 consensus_meta.cc:287] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.649s user 0.010s sys 0.002s
I20260929 14:12:17.140939 5898 raft_consensus.cc:1226] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: []
W20260929 14:12:17.297847 7678 consensus_meta.cc:287] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.657s user 0.012s sys 0.000s
I20260929 14:12:17.310336 7673 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:17.505185 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:17.179423 7464 raft_consensus.cc:2726] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:17.494056 6868 raft_consensus.cc:729] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:17.502377 7476 raft_consensus.cc:2726] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:17.504310 7682 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:17.502125 7676 raft_consensus.cc:2726] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:17.547722 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 71077us
I20260929 14:12:17.559803 5242 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:17.550943 7741 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:17.561132 5242 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:17.561234 7741 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:17.562845 5242 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:17.564227 5242 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:17.536842 7678 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:17.552707 7718 log.cc:924] Time spent T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.656s user 0.000s sys 0.001s
I20260929 14:12:17.505530 7647 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:17.564848 7682 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:17.549793 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 52630.4us
W20260929 14:12:17.567224 7734 log.cc:924] Time spent T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.270s user 0.003s sys 0.000s
I20260929 14:12:17.567636 5316 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:12:17.569849 7724 log.cc:924] Time spent T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.628s user 0.002s sys 0.002s
I20260929 14:12:17.572139 7341 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:17.572726 7341 consensus_queue.cc:1046] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260929 14:12:17.581915 5270 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:12:17.548390 7722 raft_consensus.cc:2726] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:17.587937 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:17.591596 5272 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:17.530143 7430 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:17.600720 5271 raft_consensus.cc:2454] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:17.567238 7682 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:17.698379 6868 consensus_queue.cc:260] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:17.596802 7464 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:17.577365 7321 raft_consensus.cc:3032] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Stepping down as leader of term 8
I20260929 14:12:17.596436 7676 raft_consensus.cc:2726] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:17.601714 7690 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:17.731729 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 248007us
I20260929 14:12:17.731994 7741 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:17.732600 5165 leader_election.cc:334] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.560s (SENT)
I20260929 14:12:17.732649 7741 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:17.729777 7321 raft_consensus.cc:729] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:17.733080 5165 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:17.733266 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:17.734007 7722 raft_consensus.cc:2726] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260929 14:12:17.734123 5165 leader_election.cc:334] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.564s (SENT)
I20260929 14:12:17.733839 7321 consensus_queue.cc:260] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:17.734571 5165 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:17.714003 7640 raft_consensus.cc:2741] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260929 14:12:17.735116 7321 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:17.735759 7656 raft_consensus.cc:2726] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260929 14:12:17.736702 5165 leader_election.cc:334] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 2.061s (SENT)
I20260929 14:12:17.732008 7430 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:17.733856 7738 raft_consensus.cc:2781] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260929 14:12:17.737506 5165 leader_election.cc:334] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 2.068s (SENT)
I20260929 14:12:17.737730 7738 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:17.737942 5165 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:17.573134 7725 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:17.738762 7656 raft_consensus.cc:2726] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:17.738793 7725 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:17.739986 5165 leader_election.cc:334] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473): Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 2.068s (SENT)
I20260929 14:12:17.740350 5165 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:17.741019 7656 raft_consensus.cc:2726] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:17.742875 7637 raft_consensus.cc:3032] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:17.743257 7637 raft_consensus.cc:729] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
W20260929 14:12:17.744446 7732 log.cc:924] Time spent T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.506s user 0.000s sys 0.001s
I20260929 14:12:17.744108 7637 consensus_queue.cc:260] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:17.734561 6868 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:17.555087 7649 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:17.738142 7738 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:17.606894 5268 raft_consensus.cc:1226] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:17.788714 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:17.789553 5273 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:17.590215 7647 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:17.741079 7725 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:17.727746 7464 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:17.741364 7690 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:17.785597 7637 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:12:17.806324 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 313700us
I20260929 14:12:17.746657 7741 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:17.801956 7464 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
W20260929 14:12:17.827976 7643 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.885s user 0.004s sys 0.012s
W20260929 14:12:17.854239 7643 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.907s user 0.004s sys 0.012s
I20260929 14:12:17.879707 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:17.875376 7649 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:17.888304 7647 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:17.891528 5195 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:17.892040 5195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:52182 (request call id 1689) took 1503 ms. Trace:
I20260929 14:12:17.892556 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:17.893185 5343 raft_consensus.cc:3032] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:17.893498 5343 raft_consensus.cc:729] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:17.894402 7741 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:17.894328 5343 consensus_queue.cc:260] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:17.895301 5343 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:17.894335 5901 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45248 (request call id 357) took 1225 ms. Trace:
I20260929 14:12:17.843799 7629 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:17.898001 7629 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:17.897687 5901 rpcz_store.cc:276] 0929 14:12:16.669285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:16.671778 (+ 2493us) service_pool.cc:227] Handling call
0929 14:12:17.894139 (+1222361us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54005e15d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:17.894276 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2392780672}
W20260929 14:12:17.900122 7719 log.cc:924] Time spent T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.942s user 0.005s sys 0.000s
I20260929 14:12:17.900821 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:17.901598 7721 log.cc:924] Time spent T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.960s user 0.002s sys 0.002s
I20260929 14:12:17.902074 5346 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 274) took 1189 ms. Trace:
I20260929 14:12:17.902637 5193 raft_consensus.cc:1203] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Deduplicated request from leader. Original: 2.5->[4.6-8.8] Dedup: 4.6->[6.7-8.8]
I20260929 14:12:17.903301 5193 raft_consensus.cc:1226] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9. Ops: [4.6-8.8]
I20260929 14:12:17.902452 5346 rpcz_store.cc:276] 0929 14:12:16.712598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:16.712738 (+ 140us) service_pool.cc:227] Handling call
0929 14:12:16.713162 (+ 424us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:16.713798 (+ 636us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:16.713836 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:16.714376 (+ 540us) log.cc:842] Serialized 26 byte log entry
0929 14:12:16.806350 (+ 91974us) raft_consensus.cc:1631] Marking committed up to 8
0929 14:12:16.806387 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:16.806463 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:17.899563 (+1093100us) spinlock_profiling.cc:243] Waited 31.7 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda46efbee 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551 00007fbda46f8509 00007fbda46f8499
0929 14:12:17.901795 (+ 2232us) raft_consensus.cc:1671] finished
0929 14:12:17.901814 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:17.902006 (+ 192us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33205,"raft.queue_time_us":38314,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":2870,"spinlock_wait_cycles":72887040,"thread_start_us":91637,"threads_started":1,"wal-append.queue_time_us":242982}
I20260929 14:12:17.904726 5276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41626 (request call id 338) took 1516 ms. Trace:
I20260929 14:12:17.906711 5316 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:12:17.915697 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 351499us
I20260929 14:12:17.916824 5242 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:17.905092 5276 rpcz_store.cc:276] 0929 14:12:16.387926 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:16.388100 (+ 174us) service_pool.cc:227] Handling call
0929 14:12:17.904539 (+1516439us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x7b5400323cd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:17.904651 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1437815680}
I20260929 14:12:17.892319 5195 rpcz_store.cc:276] 0929 14:12:16.388832 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:16.389011 (+ 179us) service_pool.cc:227] Handling call
0929 14:12:17.889764 (+1500753us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b5400351454. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a64da7 00007fbda99b0ee0 00007fbda30a2784 00007fbda30a2712 00007fbda30a2641 00007fbda30a25bc 00007fbda30a1872 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:17.891986 (+ 2222us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":235,"spinlock_wait_cycles":51386112}
W20260929 14:12:17.956091 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 79624.1us
I20260929 14:12:17.981840 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:17.983389 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:17.900161 7629 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:17.992029 5903 raft_consensus.cc:1226] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:17.956239 7341 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260929 14:12:17.994827 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 247866us
I20260929 14:12:17.996315 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:17.999806 7464 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:18.000946 7737 log.cc:924] Time spent T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.213s user 0.000s sys 0.000s
I20260929 14:12:17.957378 7406 consensus_queue.cc:1035] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.045s
I20260929 14:12:18.045960 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:17.963220 7676 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:18.015833 5167 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:18.016810 5871 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:18.020606 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:18.030519 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:18.032703 5343 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:18.038877 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:18.049907 5198 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:18.046291 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:18.046753 5268 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:18.051184 5164 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:18.061328 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 45669.2us
I20260929 14:12:18.061430 5274 raft_consensus.cc:1226] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:18.062306 5167 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:18.064213 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:18.064901 5267 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:18.072259 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:18.070258 5314 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:18.074271 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:18.077008 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:18.077661 5199 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:18.079241 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:18.067461 7656 raft_consensus.cc:2726] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:18.063282 7676 raft_consensus.cc:686] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:18.106943 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:18.107839 5194 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:18.072304 7690 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:18.114074 5314 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:18.117996 7760 log.cc:924] Time spent T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.056s user 0.004s sys 0.005s
I20260929 14:12:18.090281 7749 raft_consensus.cc:2781] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:18.089776 7481 raft_consensus.cc:2781] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:12:18.090178 7722 raft_consensus.cc:2726] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260929 14:12:18.089536 7678 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:18.129362 7720 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260929 14:12:18.130530 7730 log.cc:924] Time spent T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.899s user 0.005s sys 0.000s
I20260929 14:12:18.131227 7341 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:18.132246 7464 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:18.132602 7690 raft_consensus.cc:2781] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:18.163654 5194 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 7.
I20260929 14:12:18.133734 7676 consensus_queue.cc:237] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:18.136708 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 61107.6us
I20260929 14:12:18.154232 7040 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:18.170390 7341 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:18.176371 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:12:18.176774 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 44569.5us
I20260929 14:12:18.145644 7720 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:18.178668 5199 raft_consensus.cc:2454] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:12:18.158643 7412 raft_consensus.cc:2726] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:18.181466 7720 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:18.125171 7723 consensus_queue.cc:1046] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260929 14:12:18.157788 7749 raft_consensus.cc:686] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:18.182315 5165 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:18.172027 7690 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:18.195247 5314 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:18.174873 7481 raft_consensus.cc:686] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [term 4 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:18.174701 7722 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:18.208431 5314 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:18.216361 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 130309us
I20260929 14:12:18.214417 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:18.205992 7690 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:18.233680 7643 raft_consensus.cc:2726] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:18.233990 5898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45248 (request call id 354) took 1581 ms. Trace:
I20260929 14:12:18.234351 7643 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:18.234997 7771 raft_consensus.cc:3032] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:12:18.234807 7643 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:18.235492 7771 raft_consensus.cc:729] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:18.236377 7771 consensus_queue.cc:260] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:18.236912 7643 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:18.204384 7765 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:18.244490 7690 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:18.237690 7341 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:18.238618 7481 consensus_queue.cc:237] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:18.263566 7763 log.cc:924] Time spent T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260929 14:12:18.264345 7749 consensus_queue.cc:237] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:18.244959 7677 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:18.182879 7040 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:18.244364 5198 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 9.
I20260929 14:12:18.269445 5351 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 276) took 1556 ms. Trace:
I20260929 14:12:18.214080 6868 raft_consensus.cc:2781] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:18.248384 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:18.234375 5898 rpcz_store.cc:276] 0929 14:12:16.652264 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:16.667432 (+ 15168us) service_pool.cc:227] Handling call
0929 14:12:16.681280 (+ 13848us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:17.534858 (+853578us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:18.233738 (+698880us) spinlock_profiling.cc:243] Waited 449 ms on lock 0x7b540060e854. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:18.233929 (+ 191us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1033295232}
I20260929 14:12:18.217779 5346 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
W20260929 14:12:18.254830 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 46516.1us
I20260929 14:12:18.262694 7341 raft_consensus.cc:2781] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:18.210379 7722 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:18.271462 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:18.280261 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:18.283080 5276 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:18.284394 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:18.262144 5197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1703) took 1157 ms. Trace:
I20260929 14:12:18.282846 5194 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:18.272076 5351 rpcz_store.cc:276] 0929 14:12:16.713141 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:16.713396 (+ 255us) service_pool.cc:227] Handling call
0929 14:12:16.713698 (+ 302us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:16.713993 (+ 295us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:16.714017 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:16.714380 (+ 363us) log.cc:842] Serialized 26 byte log entry
0929 14:12:17.041903 (+327523us) raft_consensus.cc:1631] Marking committed up to 8
0929 14:12:17.042279 (+ 376us) log.cc:842] Serialized 14 byte log entry
0929 14:12:17.122739 (+ 80460us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:17.122805 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:17.300683 (+177878us) spinlock_profiling.cc:243] Waited 7.36 ms on lock 0x7b48000429d8. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda99147f1 00007fbda990b8e7 00007fbda990b77a 00007fbda99ff95f 00007fbda99ff627 00007fbda9a13b2c 00007fbda9a13ad8 00007fbda9a13a58 00007fbda9a13a13 00007fbda9a12cbc 00007fbda8994be6
0929 14:12:18.056049 (+755366us) spinlock_profiling.cc:243] Waited 27.7 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda46efbee 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551 00007fbda46f8509 00007fbda46f8499
0929 14:12:18.269254 (+213205us) raft_consensus.cc:1671] finished
0929 14:12:18.269276 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:18.269387 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":604125,"prepare.queue_time_us":112,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":129418,"raft.run_cpu_time_us":6637,"raft.run_wall_time_us":551413,"spinlock_wait_cycles":80673408,"thread_start_us":292463,"threads_started":1,"wal-append.queue_time_us":416523}
I20260929 14:12:18.285844 5197 rpcz_store.cc:276] 0929 14:12:17.104497 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:17.883095 (+778598us) service_pool.cc:227] Handling call
0929 14:12:17.883428 (+ 333us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:17.883640 (+ 212us) raft_consensus.cc:1520] Early marking committed up to index 4
0929 14:12:17.883943 (+ 303us) log.cc:842] Serialized 14 byte log entry
0929 14:12:18.261686 (+377743us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:18.261758 (+ 72us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:12:18.261827 (+ 69us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:18.261914 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:18.262071 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18124,"prepare.queue_time_us":678,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":34,"raft.queue_time_us":56407,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":10439,"thread_start_us":359200,"threads_started":2,"wal-append.queue_time_us":287491}
I20260929 14:12:18.340538 5346 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 7.
I20260929 14:12:18.278030 7771 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:18.278501 6868 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:18.282045 7341 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:18.348011 5351 raft_consensus.cc:1261] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260929 14:12:18.350584 7729 log.cc:924] Time spent T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 1.082s user 0.000s sys 0.002s
I20260929 14:12:18.354300 5197 raft_consensus.cc:1226] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:18.348732 5314 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:18.278323 7677 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:18.356334 7719 log.cc:924] Time spent T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.072s user 0.001s sys 0.000s
I20260929 14:12:18.266439 7765 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:18.366904 7722 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:18.355386 6868 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:18.356747 7341 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:12:18.371841 7738 consensus_meta.cc:287] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.581s user 0.008s sys 0.000s
I20260929 14:12:18.374768 7040 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:18.439894 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 45990.4us
W20260929 14:12:18.450392 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 274351us
I20260929 14:12:18.392566 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:18.392807 5351 raft_consensus.cc:1261] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260929 14:12:18.399817 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 51203.1us
I20260929 14:12:17.335609 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:18.428464 7738 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:18.433676 7641 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:18.493640 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:18.494642 5269 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:18.400319 7593 consensus_queue.cc:1046] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.137s
I20260929 14:12:18.450965 7744 consensus_queue.cc:1035] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.112s
I20260929 14:12:18.531528 5165 leader_election.cc:398] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7.
I20260929 14:12:18.450634 7745 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:18.599422 7677 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:18.533666 5268 raft_consensus.cc:3037] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:18.607489 7738 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:18.934131 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:18.603961 7720 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:18.935173 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:18.957711 5344 raft_consensus.cc:2354] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:18.935194 5196 raft_consensus.cc:1226] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:18.607934 7341 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:18.560421 7640 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:18.934938 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:18.962596 5195 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:18.776405 7690 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:18.941752 7742 consensus_queue.cc:1035] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:12:18.964365 5239 leader_election.cc:398] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:18.964726 5239 leader_election.cc:401] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260929 14:12:18.770752 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:18.943022 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 412477us
I20260929 14:12:18.966684 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:18.967454 5901 raft_consensus.cc:2354] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:18.968779 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:18.943763 7765 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:18.485848 7040 consensus_queue.cc:1046] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.474s
I20260929 14:12:18.955152 7774 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:18.487373 7641 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:18.954778 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:18.978236 5267 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:18.961316 5314 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:18.965133 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 509444us
I20260929 14:12:18.970937 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:18.598551 6868 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:18.972030 5242 leader_election.cc:398] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Vote denied by peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497) with higher term. Message: Invalid argument: T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
W20260929 14:12:18.938887 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 406658us
I20260929 14:12:18.983273 5272 raft_consensus.cc:1261] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260929 14:12:18.984319 7690 raft_consensus.cc:2781] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:18.984726 7690 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:18.981036 7641 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:18.985091 7690 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:18.986290 5342 raft_consensus.cc:1261] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:12:18.987744 7765 consensus_queue.cc:1035] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:12:18.988386 7691 raft_consensus.cc:3032] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:18.988787 7691 raft_consensus.cc:729] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:18.980427 5165 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:18.992707 5165 leader_election.cc:398] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7.
I20260929 14:12:18.993460 5165 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:18.949837 7745 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:18.998898 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:18.998404 5267 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:19.000383 5195 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:19.002998 5165 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.005559 5165 leader_election.cc:398] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:12:19.008572 5242 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.019347 7767 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:19.004604 7749 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:18.970770 7640 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:19.003547 7765 raft_consensus.cc:2781] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:19.000677 7771 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:19.025424 5165 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.028867 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.031625 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.029441 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.037988 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.039402 5200 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:19.039441 5197 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:19.044600 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.004601 7412 raft_consensus.cc:2726] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:19.019937 7749 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:19.042507 5268 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 7.
I20260929 14:12:19.047221 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:18.981184 7774 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.062500 5197 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:19.045450 5195 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:19.072265 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:19.058354 7540 raft_consensus.cc:2726] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:19.073241 5898 raft_consensus.cc:2396] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260929 14:12:19.076442 5314 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:19.031319 6868 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:19.078613 7682 raft_consensus.cc:2781] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:19.079192 7682 raft_consensus.cc:686] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:19.080149 7682 consensus_queue.cc:237] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.081893 7341 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:19.083062 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:19.083626 5272 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:18.989670 7691 consensus_queue.cc:260] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.084795 7691 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:19.026715 7765 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:19.086081 7765 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:19.091104 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:19.091725 5271 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:19.099339 5200 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:19.105077 5199 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:19.045871 7722 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:19.027493 7745 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:19.111732 5343 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:19.115435 5197 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
W20260929 14:12:19.115906 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 109168us
I20260929 14:12:19.122262 5268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41626 (request call id 367) took 1063 ms. Trace:
I20260929 14:12:19.064893 7749 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:19.122907 5272 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:19.122988 5271 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:19.038547 7701 consensus_queue.cc:1035] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
I20260929 14:12:19.130458 5165 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.065318 7738 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:19.128109 5343 raft_consensus.cc:595] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:19.130467 5268 rpcz_store.cc:276] 0929 14:12:18.059181 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:18.059373 (+ 192us) service_pool.cc:227] Handling call
0929 14:12:19.122096 (+1062723us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b54003741d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:19.122200 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1059585792}
I20260929 14:12:19.139204 5871 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:19.046882 7771 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:19.152419 5165 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.155609 5343 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:19.156414 5343 raft_consensus.cc:595] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:19.158586 7686 raft_consensus.cc:3032] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:12:19.158946 7686 raft_consensus.cc:729] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:19.147975 7754 raft_consensus.cc:2781] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:19.145256 7785 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:19.144413 7742 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:19.136351 7722 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.182783 5165 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.202042 5343 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:19.207185 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:19.199229 7720 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:19.178362 7742 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:19.215940 5165 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:19.199625 7656 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:19.188288 7754 raft_consensus.cc:686] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [term 7 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
W20260929 14:12:19.320097 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 104222us
I20260929 14:12:19.223081 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:19.229777 7783 log.cc:924] Time spent T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.092s user 0.000s sys 0.000s
I20260929 14:12:19.326575 5242 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.244508 7722 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:19.180395 7785 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:19.251550 7641 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.258411 7640 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:19.259336 7774 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:19.217561 7742 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:19.217943 7776 raft_consensus.cc:2781] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260929 14:12:19.262490 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 131119us
I20260929 14:12:19.159718 7686 consensus_queue.cc:260] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:19.264827 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:19.327711 7738 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:19.285917 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:19.287686 5097 catalog_manager.cc:5495] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 3 to 6, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 6 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:19.256636 7690 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.300392 7406 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:19.343492 5097 catalog_manager.cc:5495] T 64181b973d25469da29a63928ede710f P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 4 to 5, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 5 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:19.344142 7406 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.346174 7406 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:19.314065 5098 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52750 (request call id 222) took 6015 ms. Trace:
W20260929 14:12:19.318797 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 243073us
I20260929 14:12:19.351277 7785 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:19.203056 5343 raft_consensus.cc:595] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:19.319365 7656 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:19.325127 7754 consensus_queue.cc:237] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:19.335201 7776 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:19.320871 7649 raft_consensus.cc:2726] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:19.359160 7776 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:19.242331 7771 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:19.323037 5165 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.362039 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:19.363627 5276 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 9.
I20260929 14:12:19.329241 5347 raft_consensus.cc:1261] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:19.365984 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:19.366197 5871 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:19.366626 5268 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:19.274467 7749 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.338135 5314 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:19.346048 5097 catalog_manager.cc:5495] T 5e230699b3fb4d4d97fdb04988babce5 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 3 to 4, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 4 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:19.348263 5346 raft_consensus.cc:1226] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260929 14:12:19.303257 7765 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.355136 7690 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:19.358032 7656 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:19.369285 5098 rpcz_store.cc:276] 0929 14:12:13.307791 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:13.310638 (+ 2847us) service_pool.cc:227] Handling call
0929 14:12:19.313967 (+6003329us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:12:16.374213 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:12:16.374757 (+ 544us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:12:16.374793 (+ 36us) write_op.cc:435] Acquired schema lock
0929 14:12:16.374809 (+ 16us) tablet.cc:660] Decoding operations
0929 14:12:16.451227 (+ 76418us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:12:16.451301 (+ 74us) write_op.cc:641] Partition lock acquired for write op
0929 14:12:16.451325 (+ 24us) tablet.cc:683] Acquiring locks for 18 operations
0929 14:12:16.452562 (+ 1237us) tablet.cc:699] Row locks acquired
0929 14:12:16.452583 (+ 21us) write_op.cc:260] PREPARE: finished
0929 14:12:16.452729 (+ 146us) write_op.cc:270] Start()
0929 14:12:16.452839 (+ 110us) write_op.cc:276] Timestamp: P: 1790691136452688 usec, L: 0
0929 14:12:16.452858 (+ 19us) op_driver.cc:331] REPLICATION: starting
0929 14:12:16.497633 (+ 44775us) log.cc:842] Serialized 7055 byte log entry
0929 14:12:16.657486 (+159853us) op_driver.cc:444] REPLICATION: finished
0929 14:12:16.660628 (+ 3142us) write_op.cc:301] APPLY: starting
0929 14:12:16.660736 (+ 108us) tablet.cc:1365] starting BulkCheckPresence
0929 14:12:16.661297 (+ 561us) tablet.cc:1368] finished BulkCheckPresence
0929 14:12:16.661485 (+ 188us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:12:16.741380 (+ 79895us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:12:16.778733 (+ 37353us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=18
0929 14:12:16.778777 (+ 44us) write_op.cc:312] APPLY: finished
0929 14:12:16.779118 (+ 341us) log.cc:842] Serialized 125 byte log entry
0929 14:12:16.779270 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:16.779456 (+ 186us) write_op.cc:454] Released schema lock
0929 14:12:16.779508 (+ 52us) write_op.cc:341] FINISH: Updating metrics
0929 14:12:16.779791 (+ 283us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:16.779815 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":11154,"reported_tablets":43,"spinlock_wait_cycles":319360,"child_traces":[["op",{"apply.queue_time_us":2822,"apply.run_cpu_time_us":5919,"apply.run_wall_time_us":119640,"num_ops":18,"prepare.queue_time_us":232945,"prepare.run_cpu_time_us":5916,"prepare.run_wall_time_us":123787,"replication_time_us":204527,"thread_start_us":125298,"threads_started":2}]]}
I20260929 14:12:19.373786 5165 leader_election.cc:398] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:12:19.384125 5268 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
W20260929 14:12:19.385756 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 47725.1us
I20260929 14:12:19.386358 5314 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:19.387249 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:19.376247 7774 raft_consensus.cc:2726] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:19.389515 5316 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:19.401857 5165 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.404829 5097 catalog_manager.cc:5495] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 2 to 3, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 3 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:19.376717 7686 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:19.408980 5314 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:19.407502 7040 raft_consensus.cc:2781] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:19.407665 7593 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:19.401824 7789 raft_consensus.cc:2781] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:19.423596 7789 raft_consensus.cc:686] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:19.424428 7789 consensus_queue.cc:237] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.425351 7308 raft_consensus.cc:979] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:19.425868 7308 raft_consensus.cc:1067] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:19.430007 7308 raft_consensus.cc:979] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:19.430444 7308 raft_consensus.cc:1067] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
W20260929 14:12:19.431751 5242 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 3.116s (SENT) [suppressed 38 similar messages]
I20260929 14:12:19.431931 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:19.434316 5316 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:19.435848 5346 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:19.436369 5346 raft_consensus.cc:595] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:19.436698 5346 raft_consensus.cc:603] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:12:19.413619 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:19.437968 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:19.433367 5097 catalog_manager.cc:5495] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 6 to 8, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 8 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:19.419726 7722 consensus_queue.cc:1046] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:12:19.418816 7321 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:19.420079 7412 raft_consensus.cc:2726] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:19.422123 5165 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:19.438225 5268 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:19.420393 7040 raft_consensus.cc:686] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:19.438553 5271 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:19.376464 7738 consensus_queue.cc:1035] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260929 14:12:19.453845 5268 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:19.454980 5271 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:19.454638 5097 catalog_manager.cc:5495] T 94e36a01dad24c1193b628e9200d3c2d P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:19.458825 5165 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:19.435675 7641 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:19.464414 5198 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:12:19.481156 5198 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:19.481535 5198 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:19.485504 5197 raft_consensus.cc:1226] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:19.464927 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.465636 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.464968 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.486382 5200 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:19.467432 5192 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:12:19.467743 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:19.468175 5193 raft_consensus.cc:1226] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:19.468698 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.471477 7412 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:19.472041 7656 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.472939 7040 consensus_queue.cc:237] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.409011 7742 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.474205 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.474699 7321 raft_consensus.cc:686] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:19.501325 5097 catalog_manager.cc:5495] T 44e89147efa646d3b112a40f53a086a3 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:19.486339 5199 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:19.502375 5195 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 9.
I20260929 14:12:19.487116 5192 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:19.504312 5192 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:19.488050 5165 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.446195 7691 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:19.486419 5201 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 8
I20260929 14:12:19.494234 5200 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:19.437808 7765 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:19.498543 5342 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:19.500454 5198 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.505276 7682 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:19.505155 7640 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:19.506019 5901 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 9.
I20260929 14:12:19.508850 7682 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:19.510427 7656 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:19.510982 5198 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:19.511063 5196 raft_consensus.cc:1226] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:19.511613 7658 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:19.512080 7658 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.513032 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:19.513600 5343 raft_consensus.cc:3032] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:19.513856 5343 raft_consensus.cc:729] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:19.514472 7341 consensus_queue.cc:1046] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 14:12:19.514744 7702 consensus_queue.cc:1046] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:19.515422 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:19.494526 7406 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:19.516238 7742 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:19.516542 7406 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:19.518548 7406 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:19.510658 5316 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:19.520287 5165 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:19.514518 5343 consensus_queue.cc:260] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.494168 7649 raft_consensus.cc:2726] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:19.524799 5314 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:19.510895 7682 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:19.532253 5201 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:19.533557 7269 raft_consensus.cc:3032] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Stepping down as leader of term 8
I20260929 14:12:19.509092 7640 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.542347 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:19.536063 7702 raft_consensus.cc:3032] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Stepping down as leader of term 7
I20260929 14:12:19.543057 5901 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:19.543361 7702 raft_consensus.cc:729] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:19.543983 5316 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:19.544003 7430 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260929 14:12:19.544443 7658 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:19.544716 7430 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:19.545117 7430 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:19.542754 7640 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:19.545435 7648 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:19.545995 7648 raft_consensus.cc:686] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:19.548818 7311 raft_consensus.cc:2781] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:19.499737 7776 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.549325 7311 raft_consensus.cc:686] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:19.550166 7311 consensus_queue.cc:237] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.550808 7776 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:19.551765 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:19.548424 7749 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:19.556295 7774 raft_consensus.cc:3032] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:19.556692 7774 raft_consensus.cc:729] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:19.557421 7774 consensus_queue.cc:260] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.558558 7774 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:19.559360 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:19.559624 7686 raft_consensus.cc:979] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:19.559980 5269 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:19.560174 7686 raft_consensus.cc:1067] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Signalling peer c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:19.564718 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:19.539670 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:19.540874 7269 raft_consensus.cc:729] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:19.532827 5199 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:19.505631 7691 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:19.544196 7702 consensus_queue.cc:260] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.560233 5242 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
W20260929 14:12:19.577740 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 71113.3us
I20260929 14:12:19.580411 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.542657 7321 consensus_queue.cc:237] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.587000 5201 raft_consensus.cc:1226] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:19.525540 7742 raft_consensus.cc:2726] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:19.591349 5269 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:50776
I20260929 14:12:19.603570 5269 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:19.603978 5269 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:19.591833 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:19.582183 5192 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.595024 5165 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.605304 5198 raft_consensus.cc:1226] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:19.547087 7648 consensus_queue.cc:237] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.607517 5192 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:19.603375 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:19.615926 5165 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.588150 7722 raft_consensus.cc:2726] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:19.592861 7691 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:19.621789 5165 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.627354 5346 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:19.627607 5165 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.617019 5343 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:19.623732 5314 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:19.603073 7640 raft_consensus.cc:2781] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:19.626420 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:19.630055 5164 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:19.604861 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:19.638805 5350 raft_consensus.cc:3032] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:19.617955 7749 raft_consensus.cc:2726] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:19.646340 5350 raft_consensus.cc:729] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:19.653790 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.654801 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:19.655072 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.655757 5195 raft_consensus.cc:2354] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:19.656204 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:19.656895 5271 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:19.657491 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.657847 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:19.657501 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:19.659585 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.656855 5267 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260929 14:12:19.660647 5267 pending_rounds.cc:76] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5: Aborting all ops after (but not including) 7
I20260929 14:12:19.660742 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.661021 5267 pending_rounds.cc:97] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260929 14:12:19.661644 5267 raft_consensus.cc:2866] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:12:19.661767 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:19.662642 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.663126 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:19.664392 5270 raft_consensus.cc:1226] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:19.666332 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:19.666841 5268 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:19.626169 7742 raft_consensus.cc:2781] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:19.678962 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.680634 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.680809 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.650977 5350 consensus_queue.cc:260] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.670399 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:19.681743 5350 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:19.669777 5199 raft_consensus.cc:2354] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:19.682746 5273 raft_consensus.cc:2379] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 7: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:19.684100 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.629249 7649 raft_consensus.cc:2726] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:19.685061 5195 raft_consensus.cc:2354] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 5. Current term is 6.
I20260929 14:12:19.686115 5343 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:19.637568 7540 raft_consensus.cc:2726] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:19.638650 7640 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:19.687376 5164 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:19.687886 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:19.696283 5271 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:19.696416 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.665140 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:19.650286 7656 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:19.704500 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.698683 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.704965 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:19.699393 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.700323 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.702950 5274 raft_consensus.cc:1261] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:12:19.706660 5200 raft_consensus.cc:2354] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 5. Current term is 6.
I20260929 14:12:19.698380 5268 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:19.703300 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:19.707496 5270 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:19.707577 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:19.708647 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.709178 5192 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:19.646162 7412 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:19.651175 7701 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260929 14:12:19.704202 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:19.710803 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 57353.7us
I20260929 14:12:19.716715 5270 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:19.718029 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:12:19.717907 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 67053.4us
I20260929 14:12:19.720624 5192 raft_consensus.cc:2354] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 7. Current term is 8.
I20260929 14:12:19.677237 7690 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:19.653060 7741 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:19.646726 7652 consensus_queue.cc:1046] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.067s
I20260929 14:12:19.723250 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:19.684719 7742 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:19.724037 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.703899 7640 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:19.728638 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.722904 7412 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:19.683188 7520 consensus_queue.cc:1046] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260929 14:12:19.739192 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.752136 5200 raft_consensus.cc:2354] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:19.740638 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.741935 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.737309 5269 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.754055 5201 raft_consensus.cc:2379] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 8: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:19.721522 7649 raft_consensus.cc:2781] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:19.750568 5316 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:19.756007 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.756083 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.722137 7690 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:19.755314 5269 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:19.722532 7656 raft_consensus.cc:686] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:19.759729 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.766050 7520 raft_consensus.cc:3032] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:19.766429 7520 raft_consensus.cc:729] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:19.767170 7520 consensus_queue.cc:260] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.768393 7520 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:19.763424 7727 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260929 14:12:19.763609 7796 log.cc:924] Time spent T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.076s user 0.001s sys 0.000s
I20260929 14:12:19.775746 7430 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:19.783082 7412 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:19.787497 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:19.789938 7656 consensus_queue.cc:237] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.734122 7742 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
W20260929 14:12:19.793931 7797 log.cc:924] Time spent T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
W20260929 14:12:19.795684 7794 log.cc:924] Time spent T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260929 14:12:19.799722 7649 raft_consensus.cc:686] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:19.800915 7649 consensus_queue.cc:237] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.805817 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:19.726771 7741 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:19.807509 5167 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.808560 7741 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:19.808804 7701 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:19.809103 7741 raft_consensus.cc:2726] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:19.809533 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:19.819522 5200 raft_consensus.cc:1226] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:19.810724 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:19.837700 5167 leader_election.cc:302] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.837723 5201 raft_consensus.cc:1261] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260929 14:12:19.843847 5350 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:19.845108 5164 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:19.830557 7723 raft_consensus.cc:3032] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Stepping down as leader of term 7
I20260929 14:12:19.863134 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.866297 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.868577 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.869534 5200 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:19.871529 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.873850 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.874625 5200 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:19.883206 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:19.884141 5345 raft_consensus.cc:3032] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:19.884528 5345 raft_consensus.cc:729] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:19.884944 5195 raft_consensus.cc:2354] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:19.883396 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.885788 5201 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:19.862107 7723 raft_consensus.cc:729] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:19.764093 7690 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:19.893623 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.891453 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.825742 7269 consensus_queue.cc:260] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:19.891927 7682 consensus_queue.cc:1035] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:19.898175 7269 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
W20260929 14:12:19.899164 5242 leader_election.cc:334] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147): Timed out: RequestConsensusVote RPC to 127.3.242.65:45147 timed out after 1.518s (SENT)
I20260929 14:12:19.893028 5194 raft_consensus.cc:3037] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:19.899634 5242 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.893672 7723 consensus_queue.cc:260] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.865995 7701 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:19.906201 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:12:19.901985 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 62347.3us
I20260929 14:12:19.866602 7722 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:19.911283 7430 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:19.879890 7412 raft_consensus.cc:2781] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:19.914309 7412 raft_consensus.cc:686] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:19.915650 7412 consensus_queue.cc:237] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.918890 7540 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:19.919549 7406 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:19.919742 7691 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.919385 7540 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:19.921051 5344 raft_consensus.cc:1226] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260929 14:12:19.922262 7540 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:19.924773 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:19.925263 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:19.927201 5197 raft_consensus.cc:1226] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:19.929574 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.930408 5193 raft_consensus.cc:2362] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4. Re-sending same reply.
I20260929 14:12:19.878619 7741 raft_consensus.cc:2726] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:19.904832 5345 consensus_queue.cc:260] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:19.940608 5242 leader_election.cc:398] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:19.946089 5242 leader_election.cc:401] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260929 14:12:19.946760 7738 raft_consensus.cc:2726] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:19.947108 5194 raft_consensus.cc:1261] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260929 14:12:19.947283 5242 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.948144 7738 raft_consensus.cc:2726] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:19.948522 5242 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.949806 5242 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.950960 5242 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.940712 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.941423 5201 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 10.
W20260929 14:12:19.955596 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 47735.5us
I20260929 14:12:19.956521 5873 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.957579 5316 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:19.943317 5195 raft_consensus.cc:1261] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:12:19.964171 5342 raft_consensus.cc:595] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:19.964861 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:19.965564 5192 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:19.963481 5200 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:19.968931 5873 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.975340 5873 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.978152 5345 raft_consensus.cc:3037] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:19.965197 7738 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:19.979554 7691 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:19.980584 7379 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:19.982956 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:19.971894 7686 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader election won for term 10
I20260929 14:12:19.984174 7807 raft_consensus.cc:2781] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:12:19.901041 7593 consensus_queue.cc:1046] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260929 14:12:19.978741 7406 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260929 14:12:19.989354 5083 catalog_manager.cc:5025] ChangeConfig:REMOVE_PEER RPC for tablet f84af62485a4405f887bb8675221641e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica d886fb1eb0474d2d9f7410c4013c45e4 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:19.993551 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:19.986699 5165 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:12:19.997622 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 46740.6us
I20260929 14:12:20.007651 5344 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 9.
I20260929 14:12:20.008029 5242 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.009754 5165 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.009529 5242 leader_election.cc:398] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:20.000304 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:20.010635 5242 leader_election.cc:401] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260929 14:12:20.011933 5165 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:20.014845 5083 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 47274.3us
I20260929 14:12:20.018452 5192 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:12:19.977475 7754 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:19.976826 7723 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:20.024807 7811 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:20.024705 7649 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:12:20.024946 5165 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:19.992682 7640 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:20.036073 5165 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.033762 5164 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:12:20.038405 5267 raft_consensus.cc:1261] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260929 14:12:20.021938 5901 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:19.953517 7722 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:20.014345 7540 raft_consensus.cc:2726] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:20.022382 7321 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:20.025961 7738 raft_consensus.cc:2726] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:20.025604 7802 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:20.028992 7807 raft_consensus.cc:686] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 4 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:20.039871 7776 raft_consensus.cc:2726] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:20.035329 7649 consensus_queue.cc:1225] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer c5ec116e39144881bdf1f2cd3e923cf5 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260929 14:12:20.014652 7658 consensus_queue.cc:1035] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260929 14:12:20.015182 7742 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:20.045207 7811 raft_consensus.cc:2726] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:20.066156 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260929 14:12:20.103533 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260929 14:12:20.068216 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 82183.3us
I20260929 14:12:20.035456 7406 consensus_queue.cc:1046] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260929 14:12:20.070510 7640 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:20.077826 5345 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:20.083248 7722 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:20.054720 7686 raft_consensus.cc:686] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
W20260929 14:12:20.053390 7806 log.cc:924] Time spent T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.059s user 0.000s sys 0.001s
I20260929 14:12:20.095616 7701 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:20.054796 7749 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:20.053268 7540 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:12:20.119274 5267 raft_consensus.cc:1261] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260929 14:12:20.119696 5270 raft_consensus.cc:1261] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:20.120191 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:20.121737 5269 raft_consensus.cc:1261] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:12:20.123729 5239 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:20.131739 5193 raft_consensus.cc:1226] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:20.054769 7321 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:20.135263 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 99270.6us
I20260929 14:12:20.147470 5165 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:20.133375 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 103152us
I20260929 14:12:20.063947 7412 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260929 14:12:20.138262 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 128859us
I20260929 14:12:20.142413 5901 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 10.
I20260929 14:12:20.143399 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:20.144763 7742 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:20.152954 5242 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.171353 5242 leader_election.cc:398] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 5. Current term is 6.
I20260929 14:12:20.171808 5242 leader_election.cc:401] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260929 14:12:20.171842 5165 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.184963 5242 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.186115 5242 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.184315 7776 raft_consensus.cc:2726] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:20.158905 7817 raft_consensus.cc:2781] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:20.187693 7820 raft_consensus.cc:2726] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:20.159020 7807 consensus_queue.cc:237] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:20.203759 7690 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:20.155812 7412 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:20.211562 7412 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Advancing to term 10
W20260929 14:12:20.188689 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 132318us
I20260929 14:12:20.190683 7406 raft_consensus.cc:2726] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:20.192273 7701 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 10 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:20.194523 7321 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:20.194679 7686 consensus_queue.cc:237] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:20.187755 7742 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:20.235706 7742 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:20.237672 7742 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:20.200340 7817 raft_consensus.cc:686] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:20.239809 7817 consensus_queue.cc:237] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:20.197892 7811 raft_consensus.cc:2726] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:12:20.201777 7738 raft_consensus.cc:2726] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 5. Current term is 6.
I20260929 14:12:20.254876 7412 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:20.268498 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:20.268919 7776 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:20.284114 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:20.285097 5194 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 6
I20260929 14:12:20.284811 7776 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:20.287667 7776 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:20.269132 5200 raft_consensus.cc:1261] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 4. (term mismatch)
W20260929 14:12:20.269855 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 100739us
I20260929 14:12:20.271200 7819 raft_consensus.cc:2726] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260929 14:12:20.271808 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 55973.5us
I20260929 14:12:20.274313 5198 raft_consensus.cc:1261] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:20.358814 5165 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.288388 5200 pending_rounds.cc:76] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd: Aborting all ops after (but not including) 3
W20260929 14:12:20.315210 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 182782us
I20260929 14:12:20.363282 5200 pending_rounds.cc:97] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd: Aborting uncommitted NO_OP operation due to leader change: 4.4
I20260929 14:12:20.188835 5242 leader_election.cc:398] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 5. Current term is 6.
I20260929 14:12:20.358322 5346 raft_consensus.cc:1226] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260929 14:12:20.363689 5242 leader_election.cc:401] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260929 14:12:20.358875 5192 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:12:20.362915 5198 raft_consensus.cc:1226] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:20.363554 5200 raft_consensus.cc:2866] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:12:20.365294 5195 raft_consensus.cc:1261] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
W20260929 14:12:20.365625 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 176921us
I20260929 14:12:20.366282 5242 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.357116 5201 raft_consensus.cc:1261] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:20.367312 5242 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:20.360726 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 126073us
I20260929 14:12:20.371076 7811 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:20.371491 7811 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:20.371856 7811 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:20.373274 5899 raft_consensus.cc:1226] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 9. Current term is 10. Ops: []
I20260929 14:12:20.374809 7769 consensus_queue.cc:1035] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:12:20.380546 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:20.381234 5274 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:20.380939 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:20.385052 7702 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:20.386332 7727 consensus_queue.cc:1035] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:20.387207 5314 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:20.387946 7727 consensus_queue.cc:1035] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:12:20.389194 7727 consensus_queue.cc:1035] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:12:20.390411 7810 raft_consensus.cc:3032] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:20.390801 7810 raft_consensus.cc:729] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:20.391559 7810 consensus_queue.cc:260] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:20.392438 5276 raft_consensus.cc:1261] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:12:20.392628 7810 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:20.396327 7649 raft_consensus.cc:465] T dfe03c3eac5d42c280945fcb723aabc3 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Not starting pre-election -- already a leader
I20260929 14:12:20.400199 5194 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:20.416409 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:20.417440 5267 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:20.421594 5269 raft_consensus.cc:1261] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:20.425490 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:20.426344 5269 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:20.436127 5274 raft_consensus.cc:2454] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 10.
W20260929 14:12:20.441874 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 74613us
I20260929 14:12:20.442873 5242 leader_election.cc:398] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 7. Current term is 8.
I20260929 14:12:20.443172 5242 leader_election.cc:401] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260929 14:12:20.406705 7771 raft_consensus.cc:3032] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Stepping down as leader of term 2
W20260929 14:12:20.454326 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 55854.2us
W20260929 14:12:20.454501 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 95801.8us
I20260929 14:12:20.468717 5242 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.470938 5165 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.481025 5269 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:20.337186 7648 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:20.475492 5316 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:20.450565 7406 raft_consensus.cc:2726] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:20.495949 5871 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:20.496131 5873 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.497388 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:20.497485 5873 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:20.396620 7738 raft_consensus.cc:2726] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 5. Current term is 6.
I20260929 14:12:20.507982 5242 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.510111 5242 leader_election.cc:398] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:20.486707 7776 raft_consensus.cc:2726] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260929 14:12:20.492063 7819 raft_consensus.cc:2726] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:20.449612 7771 raft_consensus.cc:729] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:20.453934 7682 consensus_queue.cc:1035] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.057s
I20260929 14:12:20.538640 5242 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.488615 7820 raft_consensus.cc:2726] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260929 14:12:20.400365 7686 raft_consensus.cc:2726] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260929 14:12:20.473466 7640 raft_consensus.cc:2726] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 7. Current term is 8.
I20260929 14:12:20.514281 7673 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260929 14:12:20.519439 7754 raft_consensus.cc:2781] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Leader election won for term 10
I20260929 14:12:20.539564 7827 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:20.544350 5242 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.503098 6868 consensus_queue.cc:1035] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.037s
I20260929 14:12:20.552477 5242 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.503412 7765 consensus_queue.cc:1046] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260929 14:12:20.538846 7741 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:20.499423 7818 consensus_queue.cc:1035] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260929 14:12:20.514520 7733 consensus_queue.cc:1035] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260929 14:12:20.541646 7406 raft_consensus.cc:2726] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:12:20.498939 7648 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:20.560714 5242 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:20.561693 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:20.537737 7592 raft_consensus.cc:2781] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:20.549826 7828 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:20.576311 7828 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:20.576725 7828 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:20.577762 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:20.558513 7827 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:20.583606 7827 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:20.584574 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:20.564365 7754 raft_consensus.cc:686] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [term 10 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:20.551381 7820 raft_consensus.cc:2726] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:20.569361 5242 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.558409 7640 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:20.562949 7412 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 10 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:20.564004 7722 raft_consensus.cc:3032] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Stepping down as leader of term 9
I20260929 14:12:20.560492 7817 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:20.589936 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:20.568857 5345 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:20.597586 5242 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.585904 7771 consensus_queue.cc:260] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:20.563717 7741 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:20.601292 5344 raft_consensus.cc:1261] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:20.587339 7834 raft_consensus.cc:2726] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260929 14:12:20.616250 5242 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.618134 5242 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.626955 5242 leader_election.cc:398] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:20.604809 7738 raft_consensus.cc:2726] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:20.602945 7722 raft_consensus.cc:729] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:20.605683 7592 raft_consensus.cc:686] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 5 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
W20260929 14:12:20.627084 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 126601us
I20260929 14:12:20.627353 5242 leader_election.cc:401] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:12:20.614077 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:20.639631 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 164246us
I20260929 14:12:20.640898 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:20.644395 5344 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:20.646628 7834 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:20.647290 7834 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:20.648480 7754 consensus_queue.cc:237] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:20.651057 7690 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:20.650662 7722 consensus_queue.cc:260] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:20.651929 7722 raft_consensus.cc:3037] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:20.654265 7741 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:20.627298 7817 raft_consensus.cc:2726] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:20.655607 7827 consensus_queue.cc:1035] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:12:20.656349 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:20.656702 7738 raft_consensus.cc:2726] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:20.657176 5345 raft_consensus.cc:2396] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260929 14:12:20.650206 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:20.663239 5196 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:20.662914 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:20.666002 5164 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.667519 5343 raft_consensus.cc:1261] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:20.670811 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:20.628099 7406 raft_consensus.cc:2726] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:20.672848 5269 raft_consensus.cc:1203] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Deduplicated request from leader. Original: 2.5->[4.6-9.8] Dedup: 4.6->[7.7-9.8]
I20260929 14:12:20.674113 5269 raft_consensus.cc:1226] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 9. Current term is 10. Ops: [4.6-9.8]
I20260929 14:12:20.658890 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:20.657713 7771 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:20.678213 7834 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:20.678817 7648 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:20.679235 7649 consensus_queue.cc:1035] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:20.659020 7592 consensus_queue.cc:237] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:20.684438 7593 consensus_queue.cc:1035] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:20.685305 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260929 14:12:20.689968 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 45877.4us
I20260929 14:12:20.690105 7321 consensus_queue.cc:1035] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:12:20.693120 7803 consensus_queue.cc:1035] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:12:20.672371 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:20.704805 5314 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:20.708110 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:20.634366 7820 raft_consensus.cc:2726] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:20.713004 5196 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:20.713968 5275 raft_consensus.cc:2379] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 4: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:20.698714 5344 raft_consensus.cc:2454] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 10.
I20260929 14:12:20.717453 5164 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:12:20.725037 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:20.731784 5165 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:20.747140 5196 raft_consensus.cc:2379] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 8: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:20.764377 5272 raft_consensus.cc:1261] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:20.748411 5165 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.669315 7741 raft_consensus.cc:2726] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:20.744485 5242 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e
W20260929 14:12:20.767043 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 65947.5us
W20260929 14:12:20.767618 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 62911.7us
I20260929 14:12:20.779790 7828 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:20.781641 7836 log.cc:924] Time spent T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.148s user 0.000s sys 0.001s
I20260929 14:12:20.769564 7412 raft_consensus.cc:2726] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:12:20.787998 7640 raft_consensus.cc:2726] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:20.790094 5350 raft_consensus.cc:2354] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5.
W20260929 14:12:20.792199 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 74862.9us
I20260929 14:12:20.791641 5351 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 8.
I20260929 14:12:20.763827 7649 raft_consensus.cc:2781] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader election won for term 10
I20260929 14:12:20.765592 7742 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:20.805106 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:20.805471 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:20.805804 5200 raft_consensus.cc:2454] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:20.810240 5165 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.822000 5271 raft_consensus.cc:1261] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260929 14:12:20.772370 7820 raft_consensus.cc:2781] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:20.845820 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:20.746376 7841 consensus_queue.cc:1035] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.119s
I20260929 14:12:20.789474 7321 consensus_queue.cc:1046] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
I20260929 14:12:20.797672 7839 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:20.769026 7811 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:20.825181 7818 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:20.801608 7412 consensus_queue.cc:1035] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.078s
I20260929 14:12:20.901721 5316 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:20.853155 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:20.822922 7649 raft_consensus.cc:686] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [term 10 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:20.847235 7648 consensus_queue.cc:1035] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.053s
I20260929 14:12:20.904251 7520 raft_consensus.cc:2781] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:20.904771 7839 raft_consensus.cc:686] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:20.854148 7820 raft_consensus.cc:686] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:20.923975 7828 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:20.924606 7820 consensus_queue.cc:237] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:20.925534 5344 raft_consensus.cc:1261] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260929 14:12:20.930032 5239 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:20.936820 5273 raft_consensus.cc:1226] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:20.938793 5201 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260929 14:12:20.945693 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:20.946623 5197 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 5
I20260929 14:12:20.957815 7839 consensus_queue.cc:237] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:20.949084 7640 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:20.962438 7811 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:20.962932 7828 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20260929 14:12:20.965891 7853 log.cc:924] Time spent T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260929 14:12:20.966315 7833 consensus_queue.cc:1035] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:12:20.968672 5898 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260929 14:12:20.969485 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:20.970088 5267 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:20.923486 7520 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:20.997673 5197 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:21.011148 5316 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:21.014442 5267 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:21.029263 7520 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:21.041822 7789 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:21.009559 7308 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260929 14:12:21.066246 7789 raft_consensus.cc:686] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:21.067305 7789 consensus_queue.cc:237] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:21.069983 7640 raft_consensus.cc:2726] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:21.083252 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:21.085479 7801 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:21.121120 7702 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:21.087394 7801 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:21.333289 7860 log.cc:924] Time spent T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.205s user 0.000s sys 0.001s
I20260929 14:12:21.050997 7649 consensus_queue.cc:237] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:21.394295 7849 log.cc:924] Time spent T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
W20260929 14:12:21.398321 7854 log.cc:924] Time spent T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.325s user 0.001s sys 0.000s
W20260929 14:12:21.400449 7852 log.cc:924] Time spent T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.251s user 0.000s sys 0.000s
W20260929 14:12:21.411039 7848 log.cc:924] Time spent T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.336s user 0.000s sys 0.001s
I20260929 14:12:21.427740 5271 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260929 14:12:21.314497 7702 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:21.445330 7702 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:21.463196 7379 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:21.464112 7379 consensus_queue.cc:1225] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer c5ec116e39144881bdf1f2cd3e923cf5 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260929 14:12:21.467242 5273 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1849) took 1039 ms. Trace:
I20260929 14:12:21.467669 5273 rpcz_store.cc:276] 0929 14:12:20.427845 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:20.431158 (+ 3313us) service_pool.cc:227] Handling call
0929 14:12:20.431525 (+ 367us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:20.937456 (+505931us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:21.467001 (+529545us) spinlock_profiling.cc:243] Waited 505 ms on lock 0x7b5400373cd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:21.467181 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1161478912}
I20260929 14:12:21.476255 5271 raft_consensus.cc:1203] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Deduplicated request from leader. Original: 2.5->[4.6-10.9] Dedup: 4.6->[7.7-10.9]
I20260929 14:12:21.476971 5267 raft_consensus.cc:1226] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:21.476935 5271 pending_rounds.cc:76] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5: Aborting all ops after (but not including) 6
I20260929 14:12:21.478637 5271 pending_rounds.cc:97] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260929 14:12:21.478963 5271 raft_consensus.cc:2866] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260929 14:12:21.481076 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 58993.7us
I20260929 14:12:21.552562 7820 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:12:21.553412 7862 log.cc:924] Time spent T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20260929 14:12:21.553910 7801 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:21.669531 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 121420us
I20260929 14:12:21.688345 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:21.689409 5343 raft_consensus.cc:2354] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e for earlier term 9. Current term is 10.
I20260929 14:12:21.565392 7820 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:21.642275 7321 consensus_queue.cc:1046] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
I20260929 14:12:21.618204 7865 consensus_queue.cc:1046] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.137s
W20260929 14:12:21.759938 7864 log.cc:924] Time spent T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.144s user 0.001s sys 0.000s
I20260929 14:12:21.764223 7789 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:21.764724 7789 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:21.754630 7520 consensus_meta.cc:287] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.701s user 0.012s sys 0.000s
I20260929 14:12:21.768461 7520 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:21.770406 5346 raft_consensus.cc:1226] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:21.770581 7520 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:21.772430 7723 raft_consensus.cc:3032] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:12:21.772879 7723 raft_consensus.cc:729] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:21.773944 7723 consensus_queue.cc:260] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:21.775415 7861 log.cc:924] Time spent T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.317s user 0.000s sys 0.001s
I20260929 14:12:21.777338 7865 consensus_queue.cc:1225] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer c5ec116e39144881bdf1f2cd3e923cf5 log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260929 14:12:21.785543 7379 raft_consensus.cc:3032] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:21.785924 7379 raft_consensus.cc:729] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:21.787027 5870 leader_election.cc:398] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 9 pre-election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e for earlier term 9. Current term is 10.
I20260929 14:12:21.787330 5870 leader_election.cc:401] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:12:21.786752 7379 consensus_queue.cc:260] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:21.792624 7867 raft_consensus.cc:3037] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Advancing to term 10
I20260929 14:12:21.752952 7871 raft_consensus.cc:3032] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:21.867159 5269 raft_consensus.cc:1261] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260929 14:12:21.859596 7846 log.cc:924] Time spent T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.083s user 0.001s sys 0.000s
I20260929 14:12:21.889541 5345 raft_consensus.cc:1261] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260929 14:12:21.899607 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:21.903976 5198 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:21.909173 5201 raft_consensus.cc:1226] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:21.912747 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:21.913832 5201 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:22.116030 7859 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:22.129048 7859 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.131057 7822 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:22.131261 7308 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:22.131604 7822 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.131982 7308 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:22.133358 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 221823us
I20260929 14:12:22.117033 7406 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:22.135398 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:22.135546 7406 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:22.140374 7889 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:22.141166 7889 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:21.918176 7738 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:22.118419 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:22.120097 5901 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45248 (request call id 368) took 1046 ms. Trace:
I20260929 14:12:22.144426 5901 rpcz_store.cc:276] 0929 14:12:21.073877 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:21.074327 (+ 450us) service_pool.cc:227] Handling call
0929 14:12:21.083053 (+ 8726us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:21.088409 (+ 5356us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:21.088438 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:21.088865 (+ 427us) log.cc:842] Serialized 26 byte log entry
0929 14:12:21.757820 (+668955us) raft_consensus.cc:1631] Marking committed up to 8
0929 14:12:21.761403 (+ 3583us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:21.761485 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:22.119921 (+358436us) raft_consensus.cc:1671] finished
0929 14:12:22.119939 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:22.120043 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":109,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"thread_start_us":477292,"threads_started":1,"wal-append.queue_time_us":1029597}
I20260929 14:12:22.120837 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:22.146924 5269 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:22.148676 5201 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:12:22.149329 5351 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47252 (request call id 1973) took 2560 ms (client timeout 1923 ms). Trace:
W20260929 14:12:22.149763 5351 rpcz_store.cc:269] 0929 14:12:19.588474 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:19.591664 (+ 3190us) service_pool.cc:227] Handling call
0929 14:12:22.149139 (+2557475us) spinlock_profiling.cc:243] Waited 998 ms on lock 0x7b5400386154. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:22.149258 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2296125184}
I20260929 14:12:22.123790 5194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 304) took 1429 ms. Trace:
I20260929 14:12:21.862401 7871 raft_consensus.cc:729] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:21.903461 7682 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:21.878598 7379 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:22.155179 7789 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:22.156730 7882 log.cc:924] Time spent T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.404s user 0.002s sys 0.000s
W20260929 14:12:22.157680 5350 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47252 (request call id 1985) took 1570 ms (client timeout 1848 ms). Trace:
W20260929 14:12:22.158031 5350 rpcz_store.cc:269] 0929 14:12:20.587473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:20.587701 (+ 228us) service_pool.cc:227] Handling call
0929 14:12:22.157505 (+1569804us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54002305d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:22.157625 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":426109696}
I20260929 14:12:22.153723 7682 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.162158 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:22.163035 7682 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:22.163213 5270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 1939) took 1455 ms. Trace:
I20260929 14:12:22.160715 7871 consensus_queue.cc:260] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.151464 5194 rpcz_store.cc:276] 0929 14:12:20.694760 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:20.695572 (+ 812us) service_pool.cc:227] Handling call
0929 14:12:20.705980 (+ 10408us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:20.706324 (+ 344us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:20.706349 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:20.706902 (+ 553us) log.cc:842] Serialized 26 byte log entry
0929 14:12:20.829337 (+122435us) raft_consensus.cc:1631] Marking committed up to 7
0929 14:12:21.335935 (+506598us) log.cc:842] Serialized 14 byte log entry
0929 14:12:22.123423 (+787488us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:22.123520 (+ 97us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:22.123573 (+ 53us) raft_consensus.cc:1671] finished
0929 14:12:22.123590 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:22.123729 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":581841,"prepare.queue_time_us":68099,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":7860,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":18941,"thread_start_us":383670,"threads_started":2,"wal-append.queue_time_us":214417}
I20260929 14:12:22.196717 5270 rpcz_store.cc:276] 0929 14:12:20.707997 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:20.709094 (+ 1097us) service_pool.cc:227] Handling call
0929 14:12:22.161966 (+1452872us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b58001e5498. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:12:22.163135 (+ 1169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31153536}
I20260929 14:12:22.178376 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:22.221895 5197 raft_consensus.cc:1261] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:12:22.131592 7867 raft_consensus.cc:2726] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e for earlier term 9. Current term is 10.
I20260929 14:12:22.232697 5201 raft_consensus.cc:2454] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:22.216506 7818 raft_consensus.cc:465] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Not starting pre-election -- already a leader
I20260929 14:12:22.220126 7656 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:22.216786 7520 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:22.219549 7040 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:22.219870 7742 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:22.236338 7871 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:22.221487 7776 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:22.225116 7819 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:12:22.262113 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 380786us
I20260929 14:12:22.244413 7749 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:22.264565 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:22.247373 7820 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:22.253319 7889 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.239509 7741 consensus_queue.cc:1035] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260929 14:12:22.255848 7893 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:22.217658 7738 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:22.276328 7789 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:22.259299 7617 consensus_queue.cc:1035] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260929 14:12:22.282493 7888 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:22.282850 5192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1810) took 1313 ms. Trace:
I20260929 14:12:22.283053 7888 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:22.283236 5192 rpcz_store.cc:276] 0929 14:12:20.969137 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:20.969317 (+ 180us) service_pool.cc:227] Handling call
0929 14:12:20.969634 (+ 317us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:22.177089 (+1207455us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:22.177133 (+ 44us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:22.177705 (+ 572us) log.cc:842] Serialized 26 byte log entry
0929 14:12:22.218640 (+ 40935us) raft_consensus.cc:1631] Marking committed up to 8
0929 14:12:22.218688 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:22.218778 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:22.282653 (+ 63875us) raft_consensus.cc:1671] finished
0929 14:12:22.282672 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:22.282797 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1207046,"raft.queue_time_us":1250838,"spinlock_wait_cycles":18374912,"thread_start_us":40636,"threads_started":1,"wal-append.queue_time_us":64664}
I20260929 14:12:22.295056 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:22.296454 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:22.297773 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:22.298461 5351 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 7.
I20260929 14:12:22.264518 7827 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:22.295796 5345 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:22.265362 7321 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:22.303050 5276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50770 (request call id 1935) took 1631 ms. Trace:
I20260929 14:12:22.308117 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:22.305785 5276 rpcz_store.cc:276] 0929 14:12:20.671661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:20.671770 (+ 109us) service_pool.cc:227] Handling call
0929 14:12:20.673626 (+ 1856us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:20.673812 (+ 186us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:20.673837 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:20.674233 (+ 396us) log.cc:842] Serialized 26 byte log entry
0929 14:12:20.772887 (+ 98654us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:12:20.989702 (+216815us) log.cc:842] Serialized 14 byte log entry
0929 14:12:20.990101 (+ 399us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:20.990191 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:21.776413 (+786222us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b1800123820. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b45638 00007fbda1b48b11 00007fbda1b48ac9 00007fbda1b48a59 00007fbda1b48a21 00007fbda1b47d1d 00007fbdaada9204 00007fbdaada9039 00007fbda1b27dc6
0929 14:12:22.289900 (+513487us) raft_consensus.cc:1671] finished
0929 14:12:22.302851 (+ 12951us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:22.302998 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12771,"prepare.queue_time_us":36923,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":13229,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":22772,"spinlock_wait_cycles":25308544,"thread_start_us":270107,"threads_started":2,"wal-append.queue_time_us":200820}
I20260929 14:12:22.276170 7865 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:22.302682 7820 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:22.310988 7820 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:22.311133 7321 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:22.285111 7888 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:22.313324 7321 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:22.262460 7520 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:22.317850 5239 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:22.324242 5239 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:22.266762 7776 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.290076 7810 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:22.264739 7040 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:22.264951 7742 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:22.326828 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 391347us
I20260929 14:12:22.261472 7656 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:22.327509 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 99840.5us
W20260929 14:12:22.325591 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 1.35023e+06us
I20260929 14:12:22.348543 5344 raft_consensus.cc:1261] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:12:22.331012 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:22.331564 5201 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260929 14:12:22.277359 7749 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:22.333068 5200 raft_consensus.cc:1261] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260929 14:12:22.338990 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:22.314471 7741 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:22.313122 7820 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:22.316532 7865 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:22.342278 7308 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:22.340692 7810 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.302376 7789 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.343873 7895 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:22.344332 7899 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:22.364215 5192 raft_consensus.cc:1261] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260929 14:12:22.353688 5195 raft_consensus.cc:1226] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:22.266448 7819 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:22.359261 7905 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:22.374002 5242 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:22.359325 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:22.357201 7889 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:22.308472 7827 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:22.360282 7520 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:22.360297 7741 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.290514 7893 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.359443 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:22.356459 5198 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:22.365391 5197 raft_consensus.cc:1261] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260929 14:12:22.372057 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:22.362259 5164 connection.cc:232] RPC call timeout handler was delayed by -1.0437657159998253s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19230000000000003s for a response
I20260929 14:12:22.376986 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:22.380510 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 114937us
I20260929 14:12:22.364538 7412 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:22.385459 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:22.371361 7895 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:22.371174 7820 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:22.371505 7899 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:22.385872 5197 raft_consensus.cc:1226] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:22.424716 7909 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:22.425257 7894 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:22.382128 7649 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:22.425181 7911 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:22.447101 5275 raft_consensus.cc:1261] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:22.447811 5275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50770 (request call id 1943) took 1724 ms. Trace:
I20260929 14:12:22.448238 5275 rpcz_store.cc:276] 0929 14:12:20.723082 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:20.746804 (+ 23722us) service_pool.cc:227] Handling call
0929 14:12:22.446576 (+1699772us) spinlock_profiling.cc:243] Waited 11.6 ms on lock 0x7b58001e5498. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:12:22.446772 (+ 196us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:22.447488 (+ 716us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:22.447743 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":26669824}
I20260929 14:12:22.381155 7889 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.462472 7738 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:22.463474 7889 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.389073 7895 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:22.425354 7909 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.468165 7904 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:22.469126 7904 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.469995 7656 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:22.470279 7898 log.cc:924] Time spent T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.159s user 0.000s sys 0.000s
I20260929 14:12:22.384262 7865 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.459362 7753 consensus_queue.cc:1035] T f84af62485a4405f887bb8675221641e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260929 14:12:22.424366 7899 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:22.447039 7894 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.455631 7649 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:22.463485 7897 log.cc:924] Time spent T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.161s user 0.001s sys 0.000s
I20260929 14:12:22.444743 5165 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:22.456713 5316 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:22.379261 7905 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.472389 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:22.458997 7843 thread.cc:636] apply [worker] (thread pool) Time spent creating pthread: real 1.572s user 0.006s sys 0.024s
I20260929 14:12:22.461040 7893 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.473748 7738 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:22.479591 7889 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:22.479257 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:22.487460 5316 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:22.472020 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:22.494535 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:22.441049 7820 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.438269 7412 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:22.489486 7843 thread.cc:602] apply [worker] (thread pool) Time spent starting thread: real 1.707s user 0.007s sys 0.025s
I20260929 14:12:22.502239 5346 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 327) took 1399 ms. Trace:
I20260929 14:12:22.507925 5346 rpcz_store.cc:276] 0929 14:12:21.103292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:21.103567 (+ 275us) service_pool.cc:227] Handling call
0929 14:12:21.103963 (+ 396us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:21.770877 (+666914us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:22.501963 (+731086us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b54002521d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:22.502168 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1532067072}
I20260929 14:12:22.512030 7776 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.489822 7520 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260929 14:12:22.499722 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 55086.3us
I20260929 14:12:22.512425 7906 consensus_queue.cc:1046] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260929 14:12:22.515213 7412 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:22.515327 7906 consensus_queue.cc:1225] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer d886fb1eb0474d2d9f7410c4013c45e4 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260929 14:12:22.515882 7906 raft_consensus.cc:3032] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:22.516167 7906 raft_consensus.cc:729] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:22.512960 7656 raft_consensus.cc:2726] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:22.517743 7040 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:22.492544 7741 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.491925 7810 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.521617 7520 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:22.521983 7520 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:22.523834 5898 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260929 14:12:22.524215 5898 pending_rounds.cc:76] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e: Aborting all ops after (but not including) 6
I20260929 14:12:22.524542 5898 pending_rounds.cc:97] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260929 14:12:22.524829 5898 raft_consensus.cc:2866] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:12:22.500761 7738 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.497639 7893 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:22.527925 7893 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.529785 7819 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:22.508479 7889 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.532435 7889 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.455610 7911 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:22.517118 7906 consensus_queue.cc:260] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:22.534534 7906 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:22.535981 7894 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.536494 7894 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:22.512594 7776 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:22.443192 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 69268.1us
I20260929 14:12:22.489351 7879 consensus_queue.cc:1046] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260929 14:12:22.536906 7894 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.538077 7893 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.750994 7889 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:22.542774 7905 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.502985 7789 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:22.513861 7871 raft_consensus.cc:2781] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:22.493137 7908 raft_consensus.cc:2726] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260929 14:12:22.805478 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 68342.1us
W20260929 14:12:22.827716 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 216289us
W20260929 14:12:22.584689 7878 log.cc:924] Time spent T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.425s user 0.000s sys 0.002s
I20260929 14:12:22.693070 7819 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:22.829376 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 217595us
I20260929 14:12:22.832134 7883 consensus_queue.cc:1035] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260929 14:12:22.832438 7894 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.832708 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:22.829933 7841 consensus_queue.cc:1035] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260929 14:12:22.519308 7749 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:22.838341 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:22.797508 7893 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:22.835409 5316 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:22.839215 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:22.839699 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:22.830374 7738 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.839023 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:22.844254 5343 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:22.858933 7820 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.844476 7819 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.827950 7911 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:22.830984 7649 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:22.851703 7520 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.826154 7871 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:22.629779 7895 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.805688 7776 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.799811 7889 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.861475 7899 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:22.874672 7871 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:22.878252 7841 raft_consensus.cc:2781] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:22.886652 7841 raft_consensus.cc:686] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:22.891119 7889 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.870869 7893 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.887791 7841 consensus_queue.cc:237] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:22.895440 7893 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.900089 7895 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.902405 7899 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:22.934444 5164 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:22.938148 5164 leader_election.cc:398] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5.
I20260929 14:12:22.938593 5164 leader_election.cc:401] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:12:22.951382 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:22.962893 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:22.964136 7656 raft_consensus.cc:2726] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5.
I20260929 14:12:22.964735 5272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 1869) took 1893 ms. Trace:
I20260929 14:12:22.965682 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:22.967360 7656 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:22.962900 7412 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:22.968575 7412 raft_consensus.cc:686] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 8 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:22.551119 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:22.968292 7656 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.970701 7412 consensus_queue.cc:237] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:22.971400 5239 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:22.971583 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:22.951417 5276 raft_consensus.cc:1226] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:22.972416 5197 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:22.985659 5200 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:22.973069 7656 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:22.973910 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:22.996534 7810 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:22.997625 7820 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:22.998589 7820 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.001830 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:23.004729 5197 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:12:22.978924 7649 consensus_queue.cc:1035] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260929 14:12:23.009155 7818 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.007117 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.008447 7911 consensus_queue.cc:1035] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:12:22.885808 7819 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.009948 7818 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.029991 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.030225 7901 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.030603 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:22.975224 5242 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:23.036078 7738 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.036569 7895 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.037148 7738 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.037767 7895 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.040387 7899 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:23.041551 7889 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.042168 7908 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.042438 7905 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:22.965324 5272 rpcz_store.cc:276] 0929 14:12:21.070745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:21.341455 (+270710us) service_pool.cc:227] Handling call
0929 14:12:21.341775 (+ 320us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:21.351984 (+ 10209us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:21.352026 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:21.369077 (+ 17051us) log.cc:842] Serialized 26 byte log entry
0929 14:12:21.670014 (+300937us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:21.670069 (+ 55us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:21.670162 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:22.542561 (+872399us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b2400127238. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda199dcf4 00007fbda199bdfb 00007fbda1a2a559 00007fbda1a2a1e5 00007fbda1a2982d 00007fbda46d846e 00007fbda46efc0c 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551
0929 14:12:22.963916 (+421355us) raft_consensus.cc:1671] finished
0929 14:12:22.963968 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:22.964450 (+ 482us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":601,"raft.queue_time_us":1405,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":19441,"spinlock_wait_cycles":43164544,"thread_start_us":267996,"threads_started":1,"wal-append.queue_time_us":727474}
I20260929 14:12:23.043064 7908 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.043850 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.044782 5343 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:12:23.044210 7810 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.046115 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.048058 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.049423 7893 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.049695 7776 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.041628 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.042737 7889 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.042362 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:23.067655 7913 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:23.068087 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.068836 7894 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:23.071105 7920 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.069573 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:23.052112 7738 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.066608 7905 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.082172 7920 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.082301 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:23.085844 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.086877 5346 raft_consensus.cc:2379] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 7: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:23.087905 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:23.088542 5346 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:23.090288 7520 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:23.091246 7520 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:23.094826 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.068616 7913 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.093927 7901 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:23.094770 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.097755 7915 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.086721 7920 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.106328 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.106642 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:12:23.110440 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 135597us
I20260929 14:12:23.111020 5347 raft_consensus.cc:1261] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260929 14:12:23.111644 5242 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:23.106562 5345 raft_consensus.cc:2396] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260929 14:12:23.082957 7820 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:23.078171 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 107095us
I20260929 14:12:23.105227 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.121869 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:23.122152 7818 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.103659 5164 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:23.119380 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:23.130915 7901 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.131675 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:23.133610 7913 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.041556 7899 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:23.134006 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:23.136216 7899 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:23.136705 5904 raft_consensus.cc:1261] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260929 14:12:23.138700 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.139170 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:23.141373 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:23.141860 7915 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.144538 7915 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.071482 7894 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.146010 7742 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:23.146209 5873 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:23.147392 7894 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.149345 7920 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:23.132946 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:23.151242 7617 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:23.127573 5348 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:12:23.159654 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:23.163013 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 67388.4us
I20260929 14:12:23.163194 7818 raft_consensus.cc:2781] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:23.189697 7905 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.189888 7818 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:23.161175 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:23.190620 7818 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:23.126211 7738 raft_consensus.cc:2781] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:23.179706 7893 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.192483 7738 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:23.193043 7738 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:23.194530 7893 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.181865 7810 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.180034 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:23.196712 7895 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:23.197588 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 66618.5us
I20260929 14:12:23.198359 7820 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:23.198916 7656 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260929 14:12:23.200232 5239 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:23.197530 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 86101.9us
I20260929 14:12:23.201988 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.094410 7520 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:23.120206 7889 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.221788 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.227545 7908 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.249562 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.263716 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.264344 7820 raft_consensus.cc:2781] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:12:23.255764 7889 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:23.280082 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.280367 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.277383 7820 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.321586 5192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1602) took 1013 ms. Trace:
I20260929 14:12:23.322070 5192 rpcz_store.cc:276] 0929 14:12:22.330346 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:22.330466 (+ 120us) service_pool.cc:227] Handling call
0929 14:12:22.342894 (+ 12428us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:22.371943 (+ 29049us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:23.276717 (+904774us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b54003db654. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:23.321366 (+ 44649us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":48235264}
I20260929 14:12:23.322646 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.323374 5350 raft_consensus.cc:2454] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:23.323681 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:23.324513 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.324743 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.326481 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:23.312659 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.327574 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.327822 5347 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:23.321658 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.328979 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.314188 7820 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:23.330966 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.332751 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.333312 7871 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.335621 7733 consensus_queue.cc:1046] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260929 14:12:23.336797 7688 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:23.336719 5316 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:23.337288 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.338707 7827 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:23.345400 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.346563 5267 raft_consensus.cc:2354] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 7. Current term is 8.
I20260929 14:12:23.347642 7909 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.349012 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.350061 5269 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:23.329270 7889 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.351178 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.352471 7871 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:23.352810 5342 raft_consensus.cc:2354] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:23.347723 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.355685 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.356423 7700 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:23.357066 7700 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:23.359617 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:23.360379 5197 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:23.361994 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:23.362674 7890 log.cc:924] Time spent T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 1.007s user 0.003s sys 0.000s
I20260929 14:12:23.362835 5347 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:23.358675 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:23.364823 5268 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:12:23.366118 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1809) took 2431 ms. Trace:
I20260929 14:12:23.356693 5873 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:23.352525 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.359704 7700 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:23.364063 5192 raft_consensus.cc:2379] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 4: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:23.366528 5199 rpcz_store.cc:276] 0929 14:12:20.934097 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:20.934464 (+ 367us) service_pool.cc:227] Handling call
0929 14:12:20.934739 (+ 275us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:21.863123 (+928384us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:21.863178 (+ 55us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:21.863758 (+ 580us) log.cc:842] Serialized 26 byte log entry
0929 14:12:22.170871 (+307113us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:22.170908 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:22.170973 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:23.361591 (+1190618us) spinlock_profiling.cc:243] Waited 2.74 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda46efbee 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551 00007fbda46f8509 00007fbda46f8499
0929 14:12:23.365589 (+ 3998us) raft_consensus.cc:1671] finished
0929 14:12:23.365607 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:23.365911 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1234985,"raft.queue_time_us":946082,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":29961,"spinlock_wait_cycles":6293760,"thread_start_us":306801,"threads_started":1,"wal-append.queue_time_us":446148}
I20260929 14:12:23.373993 7688 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:23.374869 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 48488.5us
I20260929 14:12:23.377836 7859 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.380424 5898 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45252 (request call id 492) took 1509 ms. Trace:
I20260929 14:12:23.382309 7520 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:23.341471 7810 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260929 14:12:23.352519 7889 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.356760 5242 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:23.391705 7749 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:23.401587 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 146899us
I20260929 14:12:23.397800 5346 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:23.406261 5276 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:23.413910 5269 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:12:23.420555 5167 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:23.427453 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:23.421597 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.455406 5192 raft_consensus.cc:2379] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 6: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:23.459321 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.461433 7889 raft_consensus.cc:2726] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:23.462610 7841 raft_consensus.cc:2781] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:23.463028 7841 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:23.463390 7841 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:23.465314 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:23.465991 5199 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:23.466817 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.415139 7412 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.468098 7412 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.470686 7656 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.471463 7656 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.508091 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.490025 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.540745 5239 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:23.553157 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:23.533591 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 117376us
W20260929 14:12:23.513661 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 111315us
I20260929 14:12:23.518517 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.557062 7914 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.557296 7688 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:23.558085 5192 raft_consensus.cc:1261] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:23.557825 7914 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.489850 7827 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.560025 7914 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.560899 5271 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:23.514691 7801 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:23.462101 7741 raft_consensus.cc:2726] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:23.561889 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.562624 5271 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:23.564447 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.472573 5316 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:23.567497 7904 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.508647 7909 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.566991 5271 raft_consensus.cc:1261] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:12:23.569280 7909 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.569854 7749 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.571390 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.572187 7749 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:23.572965 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 255123us
I20260929 14:12:23.573928 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.574076 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:23.529836 7936 log.cc:924] Time spent T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.176s user 0.001s sys 0.000s
I20260929 14:12:23.573247 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.578329 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:23.578815 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.580874 5239 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:23.582641 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.586167 5197 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 9.
I20260929 14:12:23.507930 7859 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:23.590715 5242 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:23.591516 7889 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:23.589174 7776 consensus_queue.cc:1035] T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.003s
I20260929 14:12:23.598824 5239 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:23.597458 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.572587 7909 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.399246 5898 rpcz_store.cc:276] 0929 14:12:21.870740 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:21.871362 (+ 622us) service_pool.cc:227] Handling call
0929 14:12:21.877851 (+ 6489us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:22.525495 (+647644us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:23.379889 (+854394us) spinlock_profiling.cc:243] Waited 641 ms on lock 0x7b5400488354. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:23.380222 (+ 333us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1474278272}
I20260929 14:12:23.573884 7412 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.578078 7810 raft_consensus.cc:2781] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:23.517891 7738 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.598636 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.602203 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.488870 7742 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.462127 7720 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:23.604415 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.606809 7895 raft_consensus.cc:2726] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:23.607729 5167 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:23.596786 7889 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:23.559952 7827 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.612366 7893 raft_consensus.cc:2781] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:23.601823 7871 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:23.616457 5871 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:23.617844 5239 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:23.616758 7889 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:23.605711 7810 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:23.619374 7893 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:23.623391 5350 raft_consensus.cc:1261] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260929 14:12:23.623764 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 212871us
I20260929 14:12:23.622895 7895 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:23.629828 5274 raft_consensus.cc:1226] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6. Ops: []
W20260929 14:12:23.639814 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 52262.1us
I20260929 14:12:23.638129 5316 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:23.624900 7810 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:23.625967 7893 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:23.649307 5242 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:23.639014 7749 raft_consensus.cc:2726] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:23.651144 7932 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.651839 7738 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:23.652575 7945 log.cc:924] Time spent T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260929 14:12:23.653045 7406 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:23.653378 7943 log.cc:924] Time spent T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.050s user 0.000s sys 0.001s
I20260929 14:12:23.654747 7841 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.656127 7895 raft_consensus.cc:686] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:23.657168 5242 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:23.657502 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 312) took 1324 ms. Trace:
I20260929 14:12:23.647966 7700 raft_consensus.cc:2726] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:23.621323 7859 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:23.665416 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 60339us
I20260929 14:12:23.656576 7908 raft_consensus.cc:2781] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:23.661284 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:23.667604 5193 rpcz_store.cc:276] 0929 14:12:22.333297 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:22.345242 (+ 11945us) service_pool.cc:227] Handling call
0929 14:12:22.345532 (+ 290us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:22.382515 (+ 36983us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:22.382546 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:22.382582 (+ 36us) raft_consensus.cc:1631] Marking committed up to 9
0929 14:12:22.382615 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:23.657061 (+1274446us) spinlock_profiling.cc:243] Waited 36.8 ms on lock 0x7b54003dac54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:23.657125 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:23.657287 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":84676480}
I20260929 14:12:23.668354 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:23.633431 7742 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.672771 7771 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:23.665681 7905 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.670791 7908 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.673806 7406 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:23.674088 7932 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.675137 7822 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:23.675259 5167 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:23.677850 5195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 311) took 1328 ms. Trace:
W20260929 14:12:23.684163 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 58764.3us
I20260929 14:12:23.633534 7720 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:23.691953 7932 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.650228 7412 consensus_queue.cc:1035] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260929 14:12:23.666499 7738 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:23.659984 7818 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.686468 5195 rpcz_store.cc:276] 0929 14:12:22.333020 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:22.333132 (+ 112us) service_pool.cc:227] Handling call
0929 14:12:22.353471 (+ 20339us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:22.371045 (+ 17574us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:23.660557 (+1289512us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b5400146ed0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6ca09 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:23.661906 (+ 1349us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46093056}
I20260929 14:12:23.683552 7908 raft_consensus.cc:3037] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:23.700380 5194 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 6.
I20260929 14:12:23.701936 5239 leader_election.cc:398] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:23.683354 7406 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.712029 5873 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:23.709641 7905 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.714300 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.716300 5239 leader_election.cc:401] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:12:23.688813 7820 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.705390 7656 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.730516 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:23.719264 7406 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:23.714262 7776 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.718793 7909 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:23.745231 7883 raft_consensus.cc:2781] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:23.745854 7883 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:23.746275 7883 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:23.742034 7905 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:23.738969 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 53496.1us
W20260929 14:12:23.760666 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 103637us
I20260929 14:12:23.761994 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.762475 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.763223 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.753347 7932 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 7
I20260929 14:12:23.759330 5270 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:12:23.764716 7776 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:23.753338 7944 log.cc:924] Time spent T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.136s user 0.001s sys 0.000s
I20260929 14:12:23.793650 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:23.794015 7540 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:23.794523 5344 raft_consensus.cc:2379] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 4: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:23.779697 7723 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:23.794651 7540 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.796731 7932 raft_consensus.cc:2726] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:23.787607 7820 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.754261 7827 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.797449 7895 consensus_queue.cc:237] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.790406 7720 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:23.787685 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 307) took 1455 ms. Trace:
I20260929 14:12:23.791961 7818 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.725751 7738 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.793396 7771 consensus_queue.cc:1035] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:12:23.771413 7859 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:23.819608 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 46389.4us
I20260929 14:12:23.796943 7540 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.798854 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.806195 5242 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:23.810770 7776 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.811932 7742 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.802438 5196 rpcz_store.cc:276] 0929 14:12:22.332236 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:22.332335 (+ 99us) service_pool.cc:227] Handling call
0929 14:12:22.354510 (+ 22175us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:22.372611 (+ 18101us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:22.372643 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:22.372686 (+ 43us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:22.372726 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:23.787283 (+1414557us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b5400144bd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:23.787339 (+ 56us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:23.787537 (+ 198us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":41165184}
I20260929 14:12:23.824426 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:23.826834 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.828864 5350 raft_consensus.cc:2354] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:23.828401 5316 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:23.815201 7738 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.839445 5239 leader_election.cc:398] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
I20260929 14:12:23.839798 5239 leader_election.cc:401] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260929 14:12:23.839828 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.846503 7840 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:23.847121 7840 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:23.849325 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 46925.3us
I20260929 14:12:23.851011 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.831038 7859 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:23.859242 7859 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:23.860237 5351 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:23.860387 7771 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election won for term 9
I20260929 14:12:23.861047 7771 raft_consensus.cc:686] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:23.862020 7771 consensus_queue.cc:237] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.864902 6868 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:23.865449 6868 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.852535 7841 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:23.919989 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:23.856803 7893 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.849167 7840 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.939755 7893 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:23.941628 7810 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:23.998912 5192 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1838) took 1043 ms. Trace:
I20260929 14:12:23.999404 5192 rpcz_store.cc:276] 0929 14:12:22.955654 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:23.559069 (+603415us) service_pool.cc:227] Handling call
0929 14:12:23.559369 (+ 300us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:23.647022 (+ 87653us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:23.647044 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:23.650171 (+ 3127us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:23.650204 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:23.998666 (+348462us) spinlock_profiling.cc:243] Waited 76.2 ms on lock 0x7b54002cd654. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:23.998713 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:23.998847 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":175192832}
I20260929 14:12:23.942263 5351 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 1659) took 1103 ms. Trace:
I20260929 14:12:23.946307 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:23.948321 6868 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:23.860278 7908 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:23.957613 7649 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:23.981102 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:24.009127 5197 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:23.982769 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:24.009429 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 210589us
I20260929 14:12:24.001518 5351 rpcz_store.cc:276] 0929 14:12:22.838569 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:22.839487 (+ 918us) service_pool.cc:227] Handling call
0929 14:12:23.942046 (+1102559us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54003aa2d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:23.942188 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2309330816}
I20260929 14:12:23.962203 7818 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:24.011327 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:24.023027 5192 raft_consensus.cc:2379] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 4: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:24.032214 5272 raft_consensus.cc:1226] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:24.029845 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:24.043939 5192 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:24.072908 5239 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:24.075667 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1836) took 1127 ms. Trace:
I20260929 14:12:24.072254 5242 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:24.005105 7889 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:24.098932 5201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 315) took 1726 ms. Trace:
I20260929 14:12:24.108479 5201 rpcz_store.cc:276] 0929 14:12:22.372042 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:22.372151 (+ 109us) service_pool.cc:227] Handling call
0929 14:12:22.372405 (+ 254us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:22.512999 (+140594us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:22.513041 (+ 42us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:22.513082 (+ 41us) raft_consensus.cc:1631] Marking committed up to 6
0929 14:12:22.513119 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:24.098397 (+1585278us) spinlock_profiling.cc:243] Waited 140 ms on lock 0x7b540031e054. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:24.098458 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:24.098760 (+ 302us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":322938112}
I20260929 14:12:24.109871 7889 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:24.055874 7649 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:24.112247 7649 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:24.105301 7818 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:24.115033 7673 raft_consensus.cc:3032] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 8 LEADER]: Stepping down as leader of term 8
I20260929 14:12:24.115520 7673 raft_consensus.cc:729] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:24.107582 7406 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:24.118319 5901 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45232 (request call id 446) took 1772 ms. Trace:
W20260929 14:12:24.129400 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 74010.3us
I20260929 14:12:24.130381 7908 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:24.112474 5198 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52188 (request call id 1603) took 1781 ms (client timeout 1967 ms). Trace:
W20260929 14:12:24.135381 5198 rpcz_store.cc:269] 0929 14:12:22.330648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:22.330771 (+ 123us) service_pool.cc:227] Handling call
0929 14:12:24.112144 (+1781373us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b5400145fd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:24.112301 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8446848}
I20260929 14:12:24.123343 7883 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:24.134948 7540 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:24.137422 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 108493us
W20260929 14:12:24.148095 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 75264.5us
I20260929 14:12:24.151235 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:24.151971 5348 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:24.136773 7540 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:24.118440 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:24.123165 7818 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:24.158677 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 326929us
I20260929 14:12:24.126307 7810 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:24.174125 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:24.123715 7406 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:24.131906 7893 raft_consensus.cc:2726] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:24.116616 7673 consensus_queue.cc:260] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:24.229760 5192 raft_consensus.cc:1261] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260929 14:12:24.275887 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 421102us
I20260929 14:12:24.194357 5197 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
W20260929 14:12:24.452998 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 397763us
W20260929 14:12:24.409300 5200 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.004s sys 0.012s
W20260929 14:12:24.454767 5200 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.639s user 0.005s sys 0.012s
I20260929 14:12:24.446309 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:24.441094 5314 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:24.172132 7738 raft_consensus.cc:2781] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:24.474593 7673 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Advancing to term 9
W20260929 14:12:24.475541 7957 log.cc:924] Time spent T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20260929 14:12:24.474440 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:24.324468 7883 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:24.257176 7912 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.727s user 0.029s sys 0.066s
I20260929 14:12:24.319213 7859 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 3. Current term is 4.
W20260929 14:12:24.497418 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 291506us
I20260929 14:12:24.504362 7540 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:24.507375 7953 log.cc:924] Time spent T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260929 14:12:24.513584 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1847) took 1330 ms. Trace:
W20260929 14:12:24.488864 7912 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.028s user 0.034s sys 0.069s
I20260929 14:12:24.518106 5348 raft_consensus.cc:2454] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:24.514137 5193 rpcz_store.cc:276] 0929 14:12:23.183420 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:23.677107 (+493687us) service_pool.cc:227] Handling call
0929 14:12:23.677499 (+ 392us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:23.678033 (+ 534us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:23.678062 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:23.678604 (+ 542us) log.cc:842] Serialized 26 byte log entry
0929 14:12:23.930870 (+252266us) raft_consensus.cc:1631] Marking committed up to 2
0929 14:12:23.930922 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:23.931005 (+ 83us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:24.513305 (+582300us) raft_consensus.cc:1671] finished
0929 14:12:24.513332 (+ 27us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:24.513506 (+ 174us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":114,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":85,"thread_start_us":215709,"threads_started":1,"wal-append.queue_time_us":648109}
I20260929 14:12:24.474133 7738 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:24.480312 7406 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:24.515556 5195 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 320) took 1236 ms. Trace:
I20260929 14:12:24.494609 7893 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:24.525528 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:24.319664 7865 consensus_queue.cc:1046] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.305s
W20260929 14:12:24.532172 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 831856us
I20260929 14:12:24.137650 5901 rpcz_store.cc:276] 0929 14:12:22.345958 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:22.354721 (+ 8763us) service_pool.cc:227] Handling call
0929 14:12:22.358867 (+ 4146us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:22.372942 (+ 14075us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:22.375981 (+ 3039us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:22.380031 (+ 4050us) raft_consensus.cc:1631] Marking committed up to 6
0929 14:12:22.381026 (+ 995us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:24.117873 (+1736847us) spinlock_profiling.cc:243] Waited 2.77 ms on lock 0x7b54006ec154. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:24.117941 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:24.118146 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6376192}
I20260929 14:12:24.536523 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:24.528849 5195 rpcz_store.cc:276] 0929 14:12:23.278682 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:23.731248 (+452566us) service_pool.cc:227] Handling call
0929 14:12:23.731629 (+ 381us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:23.732092 (+ 463us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:23.732119 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:23.744396 (+ 12277us) log.cc:842] Serialized 26 byte log entry
0929 14:12:24.012273 (+267877us) raft_consensus.cc:1631] Marking committed up to 9
0929 14:12:24.379389 (+367116us) log.cc:842] Serialized 14 byte log entry
0929 14:12:24.379693 (+ 304us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:24.379789 (+ 96us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:24.515310 (+135521us) raft_consensus.cc:1671] finished
0929 14:12:24.515328 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:24.515507 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1220,"prepare.queue_time_us":121630,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":9472,"raft.queue_time_us":2703,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":15177,"thread_start_us":566297,"threads_started":2,"wal-append.queue_time_us":545109}
I20260929 14:12:24.526239 7738 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
W20260929 14:12:24.543553 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 367380us
I20260929 14:12:24.532975 7720 raft_consensus.cc:2726] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:24.557121 5239 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:24.559113 5343 raft_consensus.cc:1226] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:24.559365 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:24.543007 7841 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:24.553535 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:24.563230 7908 raft_consensus.cc:2781] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:24.537496 7893 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:24.590709 5899 raft_consensus.cc:2396] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260929 14:12:24.593636 7908 raft_consensus.cc:686] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:24.595124 7908 consensus_queue.cc:237] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:24.562207 7406 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:24.601507 5242 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:24.602761 7893 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:24.603427 7893 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:24.603780 7893 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:24.604113 7893 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:24.606846 7406 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:24.570864 7841 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:24.076157 5199 rpcz_store.cc:276] 0929 14:12:22.948502 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:23.554332 (+605830us) service_pool.cc:227] Handling call
0929 14:12:23.594193 (+ 39861us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:23.607440 (+ 13247us) raft_consensus.cc:1520] Early marking committed up to index 5
0929 14:12:23.825297 (+217857us) log.cc:842] Serialized 14 byte log entry
0929 14:12:24.075236 (+249939us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:24.075302 (+ 66us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:24.075368 (+ 66us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:24.075455 (+ 87us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:24.075598 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21470,"prepare.queue_time_us":313513,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":9851,"thread_start_us":357748,"threads_started":2}
I20260929 14:12:24.608865 5342 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:24.610057 7841 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:24.610110 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:24.611536 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:24.612687 5194 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 1850) took 1428 ms. Trace:
I20260929 14:12:24.613092 5194 rpcz_store.cc:276] 0929 14:12:23.184434 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:23.708644 (+524210us) service_pool.cc:227] Handling call
0929 14:12:23.719811 (+ 11167us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:23.795392 (+ 75581us) raft_consensus.cc:1520] Early marking committed up to index 9
0929 14:12:24.011946 (+216554us) log.cc:842] Serialized 14 byte log entry
0929 14:12:24.454723 (+442777us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:24.454773 (+ 50us) raft_consensus.cc:1631] Marking committed up to 9
0929 14:12:24.454819 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:24.612478 (+157659us) spinlock_profiling.cc:243] Waited 35.3 ms on lock 0x7b540031e554. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:24.612521 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:24.612633 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":322349,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":48,"raft.queue_time_us":4929,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":4299,"spinlock_wait_cycles":81169664,"thread_start_us":515499,"threads_started":2,"wal-append.queue_time_us":480618}
I20260929 14:12:24.620103 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:24.643988 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:24.713165 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 172481us
I20260929 14:12:24.647457 7895 consensus_queue.cc:1035] T 28e681a7ebc549789ec0a6f1a75c45d9 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.053s
W20260929 14:12:24.716280 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 180834us
I20260929 14:12:24.709998 7859 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:24.690879 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:24.719389 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:24.720072 5316 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:24.719161 7807 consensus_queue.cc:1046] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20260929 14:12:24.744081 7866 raft_consensus.cc:3032] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:24.744493 7866 raft_consensus.cc:729] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:24.745477 7866 consensus_queue.cc:260] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:24.746169 5268 raft_consensus.cc:1226] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:24.746971 7866 raft_consensus.cc:3037] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Advancing to term 6
W20260929 14:12:24.755648 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 230643us
W20260929 14:12:24.765381 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 222845us
W20260929 14:12:24.765908 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 87706.3us
W20260929 14:12:24.764309 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 45714.4us
I20260929 14:12:24.779444 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:24.766295 7720 raft_consensus.cc:2781] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:24.814360 5316 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:24.811992 5167 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:24.822484 5164 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:24.834456 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 111855us
I20260929 14:12:24.767437 7893 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:24.771907 7859 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:24.813390 7720 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:24.849562 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:24.850001 7869 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.737s user 0.014s sys 0.027s
W20260929 14:12:24.850425 7869 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.826s user 0.014s sys 0.028s
I20260929 14:12:24.850308 5268 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
W20260929 14:12:24.869853 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 55580.8us
I20260929 14:12:24.871915 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:24.903872 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 49214.9us
I20260929 14:12:24.905602 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:24.906708 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:24.845456 7914 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:24.910491 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:24.911056 7969 log.cc:924] Time spent T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20260929 14:12:24.854018 7720 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:24.910972 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:24.905277 7976 raft_consensus.cc:2781] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:24.823892 7933 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.085s
I20260929 14:12:24.911988 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:24.912972 7978 log.cc:924] Time spent T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260929 14:12:24.912843 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:24.913569 7971 log.cc:924] Time spent T b7924b2689dc46c98af8c4344a1e1e33 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.223s user 0.001s sys 0.000s
I20260929 14:12:24.914108 7859 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:24.914651 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:24.931574 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:24.931865 5268 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:12:24.932245 5343 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:24.844997 7749 raft_consensus.cc:2726] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:24.906235 5271 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:24.924014 5276 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:24.926865 5274 raft_consensus.cc:2379] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 7: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:24.926678 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:24.930517 5314 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:12:24.938941 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:24.939857 5269 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:24.940575 5239 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:24.921892 7914 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:24.946172 5200 raft_consensus.cc:2454] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:12:24.951506 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:24.952421 5271 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:12:24.954558 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:24.953599 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:24.956532 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:24.957196 5267 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 2.
I20260929 14:12:24.958812 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:24.959645 5276 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:24.962442 5273 raft_consensus.cc:1226] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:24.964072 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:24.964680 5194 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:24.960345 6868 consensus_queue.cc:1035] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:12:24.969394 7822 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:24.973023 7822 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:24.973436 7822 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:24.974615 7893 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:24.968976 7976 raft_consensus.cc:686] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:24.982368 7976 consensus_queue.cc:237] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:24.955601 7914 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:24.989444 7822 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:24.993607 7822 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:24.955751 7859 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:24.995353 7859 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:24.995728 7859 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:24.999048 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:25.005475 5314 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:25.006763 5314 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:25.007849 5314 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:25.010079 6868 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:25.026142 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:25.026540 5270 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:12:25.007501 7822 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:25.045275 6868 consensus_queue.cc:1035] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260929 14:12:25.050061 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 46560.8us
I20260929 14:12:25.057873 5898 raft_consensus.cc:1226] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260929 14:12:25.060606 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 51127.7us
I20260929 14:12:25.047351 7822 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:25.079737 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:25.044332 7738 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:25.084232 5342 raft_consensus.cc:1261] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260929 14:12:25.092507 5272 raft_consensus.cc:1226] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:25.073711 7822 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:25.096541 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:25.097520 5272 raft_consensus.cc:2379] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 4: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:25.098418 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:25.099469 5904 raft_consensus.cc:1261] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260929 14:12:25.101368 5097 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44250 (request call id 126) took 13171 ms (client timeout 15000 ms). Trace:
I20260929 14:12:25.104859 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:25.107321 5270 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:25.099479 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:25.107779 6868 consensus_queue.cc:1035] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:12:25.110437 5350 raft_consensus.cc:1226] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:25.116690 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:25.117982 5269 raft_consensus.cc:1226] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:25.144327 5194 raft_consensus.cc:2454] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 7.
I20260929 14:12:25.114104 6868 consensus_queue.cc:1225] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer 675b77a0a88a4bf08134281fc0cd949e log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260929 14:12:25.103204 7893 consensus_queue.cc:1035] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.022s
I20260929 14:12:25.075593 7908 consensus_queue.cc:1046] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260929 14:12:25.128177 7914 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:25.129547 7859 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:25.129333 5099 catalog_manager.cc:5495] T 6d2589e399004a6797ef987deb148b6c P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 2 to 6, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 6 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:25.130774 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:25.135489 7738 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:25.120841 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:25.201920 5267 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:25.113279 7691 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260929 14:12:25.195055 7957 log.cc:924] Time spent T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.089s user 0.000s sys 0.000s
I20260929 14:12:25.251724 5200 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:25.279196 5274 raft_consensus.cc:2354] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6.
I20260929 14:12:25.292037 5873 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:25.293095 7979 raft_consensus.cc:2781] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:25.293543 7979 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:25.293855 7979 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:25.302956 5342 raft_consensus.cc:1261] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260929 14:12:25.260645 7949 log.cc:924] Time spent T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260929 14:12:25.309480 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:25.310252 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 185863us
I20260929 14:12:25.307968 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:25.317756 5271 raft_consensus.cc:2379] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 6: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:25.314960 5873 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:25.326270 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:12:25.299293 5165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.881s (SENT) [suppressed 5 similar messages]
I20260929 14:12:25.328508 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:25.217831 7866 consensus_queue.cc:1046] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.096s
I20260929 14:12:25.338076 7859 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:25.338461 7866 raft_consensus.cc:3032] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:25.329814 7865 raft_consensus.cc:2781] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:25.338858 7866 raft_consensus.cc:729] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:25.339268 7865 raft_consensus.cc:686] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [term 7 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:25.339721 7866 consensus_queue.cc:260] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:25.340158 7865 consensus_queue.cc:237] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:25.340917 7866 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:25.341708 7893 consensus_queue.cc:1035] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 14:12:25.345592 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:25.358392 5342 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:25.351476 7720 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:25.360379 5903 raft_consensus.cc:2454] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:25.384828 5343 raft_consensus.cc:2354] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:25.372622 5099 catalog_manager.cc:5495] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 4 to 6, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 6 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:25.398698 5267 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 9.
I20260929 14:12:25.401602 5270 raft_consensus.cc:1261] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260929 14:12:25.101820 5097 rpcz_store.cc:269] 0929 14:12:11.912690 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:11.913463 (+ 773us) service_pool.cc:227] Handling call
0929 14:12:25.078367 (+13164904us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:12:19.958826 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:12:19.992296 (+ 33470us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:12:19.992336 (+ 40us) write_op.cc:435] Acquired schema lock
0929 14:12:19.992353 (+ 17us) tablet.cc:660] Decoding operations
0929 14:12:19.993296 (+ 943us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:12:19.993361 (+ 65us) write_op.cc:641] Partition lock acquired for write op
0929 14:12:19.993388 (+ 27us) tablet.cc:683] Acquiring locks for 7 operations
0929 14:12:19.994008 (+ 620us) tablet.cc:699] Row locks acquired
0929 14:12:19.994033 (+ 25us) write_op.cc:260] PREPARE: finished
0929 14:12:19.994182 (+ 149us) write_op.cc:270] Start()
0929 14:12:19.994293 (+ 111us) write_op.cc:276] Timestamp: P: 1790691139994132 usec, L: 0
0929 14:12:19.994315 (+ 22us) op_driver.cc:331] REPLICATION: starting
0929 14:12:20.045338 (+ 51023us) log.cc:842] Serialized 2814 byte log entry
0929 14:12:20.782497 (+737159us) op_driver.cc:444] REPLICATION: finished
0929 14:12:22.312028 (+1529531us) spinlock_profiling.cc:243] Waited 8.53 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda1b275fe 00007fbdaa95ea39 00007fbda1b448bb 00007fbda1b42cbf 00007fbda1b44e22 00007fbda847d410
0929 14:12:22.459418 (+147390us) write_op.cc:301] APPLY: starting
0929 14:12:22.459503 (+ 85us) tablet.cc:1365] starting BulkCheckPresence
0929 14:12:22.459699 (+ 196us) tablet.cc:1368] finished BulkCheckPresence
0929 14:12:22.459721 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:12:22.460463 (+ 742us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:12:22.460672 (+ 209us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0929 14:12:22.460719 (+ 47us) write_op.cc:312] APPLY: finished
0929 14:12:22.460992 (+ 273us) log.cc:842] Serialized 59 byte log entry
0929 14:12:23.868860 (+1407868us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda1b275fe 00007fbdaa95ea39 00007fbda1b448bb 00007fbda1b42cbf 00007fbda1b44e22 00007fbda46ee5bb
0929 14:12:24.558727 (+689867us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:24.598384 (+ 39657us) write_op.cc:454] Released schema lock
0929 14:12:24.598477 (+ 93us) write_op.cc:341] FINISH: Updating metrics
0929 14:12:24.598921 (+ 444us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:24.598957 (+ 36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":608082,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":1659596,"apply.run_cpu_time_us":108079,"apply.run_wall_time_us":2140046,"num_ops":7,"prepare.queue_time_us":268212,"prepare.run_cpu_time_us":13508,"prepare.run_wall_time_us":354476,"raft.queue_time_us":201431,"raft.run_cpu_time_us":31961,"raft.run_wall_time_us":1731652,"replication_time_us":788084,"spinlock_wait_cycles":47250048,"thread_start_us":4144296,"threads_started":5,"wal-append.queue_time_us":2495149,"wal-append.run_cpu_time_us":40205,"wal-append.run_wall_time_us":1165580}]]}
I20260929 14:12:25.412020 5272 raft_consensus.cc:1226] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:25.413357 5342 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:25.415510 5314 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:25.409905 5268 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:25.418731 5239 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:25.394429 7822 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:25.421737 7771 consensus_queue.cc:1046] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:12:25.423099 5314 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:25.423683 7822 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:25.424206 5314 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:25.425412 7907 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:25.425987 7907 raft_consensus.cc:2726] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:25.423230 7810 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:25.428153 7810 raft_consensus.cc:686] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:25.429361 7810 consensus_queue.cc:237] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:25.434988 7771 raft_consensus.cc:3032] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:25.435494 7771 raft_consensus.cc:729] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:25.436355 7771 consensus_queue.cc:260] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:25.442480 7977 consensus_queue.cc:1046] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:12:25.446914 7771 raft_consensus.cc:3037] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:25.450923 7720 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:25.451826 7914 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:25.424279 7822 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:12:25.481980 5906 raft_consensus.cc:1261] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260929 14:12:25.483646 5099 catalog_manager.cc:5495] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260929 14:12:25.503763 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 203831us
W20260929 14:12:25.516508 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 177308us
I20260929 14:12:25.535557 6868 consensus_queue.cc:1046] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:25.544958 5165 leader_election.cc:398] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 7. Current term is 8.
I20260929 14:12:25.546224 5165 leader_election.cc:401] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:12:25.548462 6868 raft_consensus.cc:3032] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:25.549033 6868 raft_consensus.cc:729] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:25.535557 5316 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:25.553189 6868 consensus_queue.cc:260] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:25.559144 6868 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:25.564064 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:25.565002 5099 catalog_manager.cc:5495] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
W20260929 14:12:25.566756 5899 consensus_meta.cc:287] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e: Time spent flushing consensus metadata: real 0.600s user 0.009s sys 0.000s
I20260929 14:12:25.567137 5899 raft_consensus.cc:2454] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 9.
W20260929 14:12:25.568058 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 257822us
I20260929 14:12:25.568611 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:25.569192 5272 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:12:25.571086 7995 log.cc:924] Time spent T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.195s user 0.001s sys 0.000s
W20260929 14:12:25.571655 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 156773us
I20260929 14:12:25.573817 7742 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:25.574354 7917 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:25.574962 7917 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:25.576536 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:25.577244 5350 raft_consensus.cc:2354] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6.
I20260929 14:12:25.578238 7827 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 8
I20260929 14:12:25.574371 7742 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:25.578280 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 48057.3us
I20260929 14:12:25.582055 5165 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:25.582868 7914 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:25.583227 7914 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:25.583552 7914 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:25.558054 7979 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:25.587232 5898 raft_consensus.cc:1226] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260929 14:12:25.587392 7841 raft_consensus.cc:3032] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:25.587796 7841 raft_consensus.cc:729] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:25.567350 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:25.566551 5099 catalog_manager.cc:5495] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:25.597934 5165 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:25.602234 5164 leader_election.cc:398] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6.
I20260929 14:12:25.603435 5164 leader_election.cc:401] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260929 14:12:25.603747 5346 raft_consensus.cc:1226] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 7. Current term is 8. Ops: []
W20260929 14:12:25.608875 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 73752.5us
I20260929 14:12:25.606366 7982 raft_consensus.cc:2726] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:25.612262 5165 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:25.588594 7841 consensus_queue.cc:260] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:25.610522 5316 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:25.614538 5165 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:25.620087 6868 raft_consensus.cc:2726] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260929 14:12:25.626842 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:25.619434 7765 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:25.637032 5239 leader_election.cc:398] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:25.637430 5239 leader_election.cc:401] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260929 14:12:25.639179 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:25.639577 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:25.639801 5196 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:25.606621 7917 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:25.642583 7933 raft_consensus.cc:3032] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:25.642953 7933 raft_consensus.cc:729] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:25.643778 7933 consensus_queue.cc:260] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:25.644791 7933 raft_consensus.cc:3037] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:25.626598 7979 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:25.624267 7981 raft_consensus.cc:2781] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:25.645723 7981 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:25.646063 7981 raft_consensus.cc:3037] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:25.653796 5099 catalog_manager.cc:5495] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 2 to 3, leader changed from 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 3 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:25.562314 7869 consensus_queue.cc:1035] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.101s
I20260929 14:12:25.667531 5272 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:25.669309 7841 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:25.681407 7321 raft_consensus.cc:2741] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260929 14:12:25.687860 7996 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:25.688555 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 51672.6us
I20260929 14:12:25.722358 7893 raft_consensus.cc:2726] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:25.753729 5196 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
W20260929 14:12:25.762416 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 64355.9us
I20260929 14:12:25.698056 7742 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:25.779618 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 226636us
I20260929 14:12:25.764189 7827 raft_consensus.cc:2726] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 7. Current term is 8.
I20260929 14:12:25.785923 5242 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:25.768797 7765 raft_consensus.cc:2726] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6.
I20260929 14:12:25.771195 7649 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:25.737000 7996 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:25.805408 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 189556us
I20260929 14:12:25.806334 5316 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:25.840016 7827 consensus_queue.cc:1046] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260929 14:12:25.846556 7996 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:25.842193 7406 raft_consensus.cc:2781] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election won for term 9
I20260929 14:12:25.847558 7406 raft_consensus.cc:686] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:25.848242 7406 consensus_queue.cc:237] T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:25.852739 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 66936.3us
I20260929 14:12:25.853329 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:25.738947 7981 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:25.858287 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 52074.8us
I20260929 14:12:25.862057 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:25.865388 5269 raft_consensus.cc:1226] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260929 14:12:25.856173 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:25.868819 7742 raft_consensus.cc:3032] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Stepping down as leader of term 7
I20260929 14:12:25.880971 5192 raft_consensus.cc:1261] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260929 14:12:25.887409 5272 raft_consensus.cc:1261] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260929 14:12:25.877043 7742 raft_consensus.cc:729] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:25.870651 7321 raft_consensus.cc:2781] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:25.892248 5199 raft_consensus.cc:1261] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260929 14:12:25.896814 7321 raft_consensus.cc:686] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:25.868822 7311 consensus_queue.cc:1046] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260929 14:12:25.897547 7311 consensus_queue.cc:1225] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer 675b77a0a88a4bf08134281fc0cd949e log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260929 14:12:25.896927 7742 consensus_queue.cc:260] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:25.897707 7321 consensus_queue.cc:237] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:25.898833 7742 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:25.857602 7996 consensus_queue.cc:1035] T 520249ed5be343dcb30c86d3396491a4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260929 14:12:25.908643 7976 consensus_queue.cc:1035] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260929 14:12:25.919097 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 304649us
I20260929 14:12:25.919904 5165 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:25.884783 7914 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:25.931339 5347 raft_consensus.cc:1261] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:25.931911 7649 raft_consensus.cc:2741] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260929 14:12:25.965698 5314 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:25.977145 5165 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:25.936347 7977 consensus_queue.cc:1035] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.030s
I20260929 14:12:25.988145 5165 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:25.974236 7822 raft_consensus.cc:2726] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260929 14:12:25.996716 7917 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:25.997130 5165 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:25.998651 7765 raft_consensus.cc:2726] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:25.944212 7893 consensus_queue.cc:1035] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260929 14:12:26.002296 5165 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:26.007589 5165 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:26.010512 5165 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:26.018324 5898 raft_consensus.cc:1226] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260929 14:12:26.018532 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:25.993661 8011 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:26.035436 5273 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260929 14:12:26.035866 5273 pending_rounds.cc:76] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5: Aborting all ops after (but not including) 6
I20260929 14:12:26.010165 7981 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:26.013643 7917 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:26.036152 5273 pending_rounds.cc:97] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260929 14:12:26.020673 7827 raft_consensus.cc:2781] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:26.040061 5273 raft_consensus.cc:2866] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:12:25.993907 7649 consensus_queue.cc:1046] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
W20260929 14:12:26.051766 8006 log.cc:924] Time spent T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.148s user 0.000s sys 0.001s
I20260929 14:12:26.059500 7822 consensus_queue.cc:1035] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260929 14:12:26.069175 7649 raft_consensus.cc:3032] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:26.069757 7649 raft_consensus.cc:729] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:26.070708 7649 consensus_queue.cc:260] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:26.035949 8011 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:26.080365 8011 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:26.084394 8008 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260929 14:12:26.085892 7914 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:26.024511 5347 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:26.038740 7992 consensus_queue.cc:1046] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260929 14:12:26.087149 5347 raft_consensus.cc:595] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:26.087487 7992 consensus_queue.cc:1225] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer 675b77a0a88a4bf08134281fc0cd949e log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260929 14:12:26.094050 7977 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:26.058633 7827 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:26.059814 7982 raft_consensus.cc:2781] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260929 14:12:26.115556 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 105035us
I20260929 14:12:26.128365 5347 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:26.129716 5347 raft_consensus.cc:595] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:26.128039 5165 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:26.131206 7649 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:26.143194 5165 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:26.094534 7977 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:26.143159 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 104690us
I20260929 14:12:26.112187 7827 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:26.164536 7914 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:26.169463 7914 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:26.169881 7914 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:26.165303 7771 raft_consensus.cc:3032] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:26.171430 7771 raft_consensus.cc:729] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:26.172226 7771 consensus_queue.cc:260] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:26.173427 7771 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:26.174829 7977 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:26.178704 8024 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:26.183104 7914 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:26.189637 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 46518.5us
I20260929 14:12:26.201817 5165 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:26.127082 7982 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:26.214161 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 204659us
I20260929 14:12:26.218232 5165 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:26.219398 5165 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:26.220402 5165 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:26.216661 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:26.222043 5165 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:26.228343 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:26.217990 7982 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:26.237255 5165 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:26.241369 5165 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:26.227263 8028 raft_consensus.cc:2726] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:26.225348 7917 raft_consensus.cc:2781] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:12:26.225178 7981 raft_consensus.cc:2781] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:26.225170 7765 raft_consensus.cc:2781] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:26.240650 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:26.245617 8024 raft_consensus.cc:2726] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:26.333998 5899 raft_consensus.cc:2379] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 9: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:26.334012 8024 raft_consensus.cc:2726] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260929 14:12:26.337332 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 146890us
W20260929 14:12:26.326952 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 82852.5us
I20260929 14:12:26.328743 7649 raft_consensus.cc:2781] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:12:26.323908 7917 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:26.337528 7765 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:26.337805 7981 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:26.242434 8025 consensus_queue.cc:1046] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.108s
I20260929 14:12:26.346031 7649 raft_consensus.cc:686] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
W20260929 14:12:26.351930 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 110633us
I20260929 14:12:26.340857 7960 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:26.354429 5343 raft_consensus.cc:2454] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:26.352510 5346 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:26.345392 7917 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:26.518384 7960 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:26.375198 7914 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:26.499562 5165 leader_election.cc:398] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6.
I20260929 14:12:26.522055 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.522964 5199 raft_consensus.cc:2396] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260929 14:12:26.523432 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.524673 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.522326 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:26.389281 7981 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:26.356412 7765 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:26.377753 8023 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260929 14:12:26.499157 8027 log.cc:924] Time spent T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29: Append to log took a long time: real 0.091s user 0.001s sys 0.001s
I20260929 14:12:26.528743 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.538671 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.538887 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.539288 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.539794 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.545468 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.547417 7960 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:26.552728 8029 log.cc:924] Time spent T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20260929 14:12:26.535866 5197 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:26.509487 7649 consensus_queue.cc:237] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:26.523509 7959 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:26.588318 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 363788us
I20260929 14:12:26.586074 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.589553 8030 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:12:26.605888 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 376661us
I20260929 14:12:26.606324 8030 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:26.608876 8030 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:26.612246 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:26.613111 8039 raft_consensus.cc:3032] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 8 LEADER]: Stepping down as leader of term 8
I20260929 14:12:26.641832 8039 raft_consensus.cc:729] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 8 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:26.641881 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.642738 8039 consensus_queue.cc:260] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:26.643457 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:26.644304 8039 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:26.620040 7827 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:26.647399 7827 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:26.620863 7959 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:26.650256 7959 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:26.621768 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:26.622393 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.659086 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:26.659801 5193 raft_consensus.cc:2379] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 5: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:26.660557 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:26.624291 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.662076 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:26.663100 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:26.664047 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.569917 8023 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:26.625414 5198 raft_consensus.cc:1226] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260929 14:12:26.629878 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.665428 8023 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:26.666078 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.667218 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.668203 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.544962 7917 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:26.669230 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.630304 7723 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:26.670233 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.670101 7723 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:26.670749 7917 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:26.672106 7723 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:26.630796 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.632833 7982 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:26.673589 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.635663 8025 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:26.673977 7981 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:26.674369 8025 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:26.676008 7981 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:26.676054 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.676401 8025 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:26.629096 8011 consensus_meta.cc:287] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.548s user 0.007s sys 0.001s
I20260929 14:12:26.677059 8011 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:26.678819 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:26.679117 8011 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:26.679842 5200 raft_consensus.cc:1226] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:26.680090 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.682745 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.683754 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.683830 5242 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:26.661159 5193 raft_consensus.cc:2379] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 5: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:26.685683 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:12:26.686971 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 80105.8us
I20260929 14:12:26.686904 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:26.664991 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:26.694216 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 160047us
I20260929 14:12:26.698325 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.690614 5314 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:26.702181 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.706696 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.711613 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.681200 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:26.719935 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.721061 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:26.723978 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.720997 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.725019 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.730890 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.734701 5196 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:26.735981 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.729807 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.737869 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:26.739122 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:26.740995 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:26.741664 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:26.731557 5197 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 2.
I20260929 14:12:26.743292 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:26.684695 8038 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:26.744699 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.753957 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:26.755577 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:26.755973 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 72207.5us
I20260929 14:12:26.756350 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:26.756932 5242 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:26.758415 5198 raft_consensus.cc:2354] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:26.759342 5197 raft_consensus.cc:2379] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 7: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:26.764743 5199 raft_consensus.cc:1261] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260929 14:12:27.026795 5273 raft_consensus.cc:1261] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:12:27.060614 5197 raft_consensus.cc:1261] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260929 14:12:27.002385 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:27.001353 5194 raft_consensus.cc:2396] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260929 14:12:26.745381 7996 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
W20260929 14:12:27.065393 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 308515us
I20260929 14:12:26.917848 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:26.944715 5200 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:26.965893 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:26.953339 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:27.073611 5192 raft_consensus.cc:1261] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:27.037730 5164 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:27.072188 5242 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:26.785930 8038 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:27.077340 5242 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.078656 5242 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.079903 5242 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.063908 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:26.780579 8041 log.cc:924] Time spent T e55a9c8ae48e44c59fee506e539b3f90 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.088s user 0.000s sys 0.001s
I20260929 14:12:27.084612 5873 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.092357 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:27.081475 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 43872.2us
I20260929 14:12:26.963450 7841 raft_consensus.cc:979] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:26.760027 7691 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:27.084714 8024 raft_consensus.cc:2781] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:27.087174 8038 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:27.087414 7933 raft_consensus.cc:2726] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:26.745684 7321 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:27.084532 7406 raft_consensus.cc:2726] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:26.775460 8021 raft_consensus.cc:2726] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:27.009931 7742 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:26.749547 7977 consensus_queue.cc:1046] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.337s
I20260929 14:12:27.076993 8019 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:27.111367 7673 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:27.003296 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:27.095351 5164 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:27.018945 7981 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:26.773093 7879 raft_consensus.cc:3037] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:27.100675 5870 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.102080 5345 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:27.098357 7903 raft_consensus.cc:2726] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260929 14:12:27.008628 7765 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.097291 7982 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:27.111467 7933 raft_consensus.cc:2726] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260929 14:12:27.118506 5345 raft_consensus.cc:595] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:27.120468 5870 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.108057 7976 raft_consensus.cc:1067] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Signalling peer c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:27.121704 8038 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:27.111474 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:27.128721 5316 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:27.107905 8024 raft_consensus.cc:686] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:27.122012 8043 raft_consensus.cc:2726] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:27.139312 5200 raft_consensus.cc:1226] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:27.119184 7673 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:27.104028 7691 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.130460 7903 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:27.142494 5164 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.155093 5198 raft_consensus.cc:1226] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:27.158283 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:27.158478 5192 raft_consensus.cc:1261] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260929 14:12:27.108605 7321 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.113636 8023 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.139894 8038 raft_consensus.cc:686] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:27.160310 5198 raft_consensus.cc:1226] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 2. Current term is 3. Ops: []
I20260929 14:12:27.117686 7742 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:27.161379 5200 raft_consensus.cc:1226] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:27.161509 5351 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:27.165937 5351 raft_consensus.cc:595] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:27.165999 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:27.166343 5351 raft_consensus.cc:603] T 6d2589e399004a6797ef987deb148b6c P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:12:27.164098 5194 raft_consensus.cc:1226] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:27.167428 8024 consensus_queue.cc:237] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.168200 7682 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:27.169445 8038 consensus_queue.cc:237] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.175228 5192 raft_consensus.cc:1226] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:27.159054 8039 raft_consensus.cc:2726] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:27.126283 7981 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:27.148742 7977 consensus_queue.cc:1035] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
W20260929 14:12:27.211360 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 114889us
I20260929 14:12:27.212522 5197 raft_consensus.cc:1226] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 7. Current term is 8. Ops: []
W20260929 14:12:27.223477 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 143605us
I20260929 14:12:27.224283 5242 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.187685 7869 consensus_queue.cc:1046] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
W20260929 14:12:27.236426 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 87061.9us
I20260929 14:12:27.238960 5349 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:27.239610 5349 raft_consensus.cc:595] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
W20260929 14:12:27.269179 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 162225us
I20260929 14:12:27.269834 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260929 14:12:27.275265 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:27.281544 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 181376us
I20260929 14:12:27.280737 5349 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 7.
I20260929 14:12:27.240767 7673 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:27.277911 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:12:27.283282 5870 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:27.286732 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 62519.9us
I20260929 14:12:27.287670 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:27.288102 5242 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.268766 7765 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:27.292533 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:27.235432 7771 consensus_queue.cc:1046] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260929 14:12:27.290988 5268 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:50776
I20260929 14:12:27.305373 5268 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:27.305812 5268 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:27.296607 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:27.297168 8019 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:27.298070 7879 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:27.298601 5242 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.280311 8050 raft_consensus.cc:2726] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:27.323767 5242 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:27.282377 8047 log.cc:924] Time spent T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.147s user 0.000s sys 0.001s
I20260929 14:12:27.299803 5276 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 6.
I20260929 14:12:27.289182 7869 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:27.324730 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 119855us
I20260929 14:12:27.338160 5268 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.344849 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:27.345928 5276 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:27.306079 7959 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:27.350718 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:27.343816 7742 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:27.341719 7321 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:27.350152 7959 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:27.353184 7959 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:27.321702 7903 consensus_queue.cc:1035] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260929 14:12:27.360997 7959 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:27.363014 7959 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 8 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:27.341485 7869 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.368916 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:27.369776 5276 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 8
I20260929 14:12:27.351442 7933 raft_consensus.cc:2726] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:27.375762 8048 raft_consensus.cc:3032] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:27.376243 8048 raft_consensus.cc:729] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:27.377324 8048 consensus_queue.cc:260] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.378808 8048 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:27.381462 7822 consensus_queue.cc:1046] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:12:27.381793 7771 consensus_queue.cc:1046] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:27.382571 7907 consensus_queue.cc:1046] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:27.383615 7907 raft_consensus.cc:3032] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:12:27.383981 7907 raft_consensus.cc:729] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:27.384330 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:27.385592 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:27.351601 8057 consensus_queue.cc:1035] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260929 14:12:27.386602 7720 consensus_queue.cc:1046] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:12:27.386940 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:27.387585 5194 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:27.344391 8023 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:27.393488 5268 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:27.349848 7691 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:27.366178 8021 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 14:12:27.365720 7869 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:27.403290 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:12:27.405517 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 101050us
I20260929 14:12:27.410641 5167 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.391620 7682 raft_consensus.cc:3032] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:27.388911 7742 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:27.384759 7907 consensus_queue.cc:260] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.426699 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:27.428323 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:27.419682 8023 raft_consensus.cc:2726] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260929 14:12:27.435235 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 111533us
I20260929 14:12:27.437493 5242 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.421911 7682 raft_consensus.cc:729] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:27.445606 5242 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.448196 5242 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.449515 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:27.450106 5347 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:27.436429 7691 raft_consensus.cc:3032] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:27.440249 8021 raft_consensus.cc:2726] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:27.456868 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:27.466287 5269 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:27.457988 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:27.467129 5268 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:27.451716 7933 raft_consensus.cc:2726] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:27.464250 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:27.439046 7981 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:27.455967 5242 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.458451 5314 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:27.424934 7742 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:27.479100 5242 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.480422 5242 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.457165 8061 raft_consensus.cc:2726] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:27.460337 7691 raft_consensus.cc:729] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:27.514077 7691 consensus_queue.cc:260] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.515427 7691 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:27.492936 7822 raft_consensus.cc:2781] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260929 14:12:27.496016 8064 log.cc:924] Time spent T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260929 14:12:27.496346 5314 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:27.497287 8051 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:27.498031 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:27.548045 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 113334us
I20260929 14:12:27.548523 5276 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 8.
I20260929 14:12:27.507488 7682 consensus_queue.cc:260] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.510313 8021 raft_consensus.cc:2726] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260929 14:12:27.481123 7933 raft_consensus.cc:2726] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:27.492147 7742 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:27.552011 5316 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:27.555583 7869 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:27.555955 7869 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:27.556268 7869 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:27.557461 8072 consensus_queue.cc:1046] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:27.521008 5347 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 8.
I20260929 14:12:27.524173 7822 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:27.531023 8058 log.cc:924] Time spent T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260929 14:12:27.480849 7907 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:27.538443 7841 raft_consensus.cc:3032] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Stepping down as leader of term 7
I20260929 14:12:27.599174 5870 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:27.547015 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 50764.2us
I20260929 14:12:27.548636 5343 raft_consensus.cc:2454] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:27.490104 5242 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.550776 8060 raft_consensus.cc:2781] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:27.563433 5197 raft_consensus.cc:1226] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:27.569730 5196 raft_consensus.cc:1226] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:27.519529 7996 raft_consensus.cc:2726] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:27.578455 5351 raft_consensus.cc:3032] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Stepping down as leader of term 7
W20260929 14:12:27.602945 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 112913us
I20260929 14:12:27.607338 5164 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:27.610904 5242 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.612404 5242 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.618010 5347 raft_consensus.cc:1261] T 4fc85ccb35394b2d800cea4f13552756 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:12:27.618130 5242 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.619421 5242 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.610533 5351 raft_consensus.cc:729] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:27.621932 5242 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.623430 5242 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.625509 5269 raft_consensus.cc:1226] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:27.577117 7822 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:27.629103 6868 consensus_queue.cc:1046] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:12:27.568225 8072 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:27.620587 5351 consensus_queue.cc:260] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.635483 5351 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:27.608249 8021 raft_consensus.cc:2726] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:27.608103 8060 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:27.598397 7841 raft_consensus.cc:729] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:27.622967 8038 raft_consensus.cc:2781] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260929 14:12:27.638998 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 167562us
I20260929 14:12:27.619472 7700 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:27.622826 7933 raft_consensus.cc:2726] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:12:27.636102 5242 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:12:27.645675 5167 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.626637 8061 raft_consensus.cc:2726] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:27.621402 8051 raft_consensus.cc:2726] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260929 14:12:27.626183 8050 raft_consensus.cc:2726] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:27.650774 5242 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:27.707621 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 57179us
I20260929 14:12:27.709419 5242 leader_election.cc:398] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5.
I20260929 14:12:27.637037 7996 raft_consensus.cc:2726] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:27.637037 7979 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:27.626911 8023 consensus_queue.cc:1035] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.025s
I20260929 14:12:27.645051 8060 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:27.630322 8019 raft_consensus.cc:2726] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:27.713757 5242 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.644209 8038 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:27.650619 7879 raft_consensus.cc:3032] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:27.671728 7700 raft_consensus.cc:686] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [term 8 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:27.667985 8021 raft_consensus.cc:2781] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260929 14:12:27.737703 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 86529.1us
W20260929 14:12:27.708225 8058 log.cc:924] Time spent T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260929 14:12:27.640087 8072 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.752341 5242 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.726482 8043 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:27.731076 8068 raft_consensus.cc:2726] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260929 14:12:27.766846 7700 consensus_queue.cc:237] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:27.769384 8043 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:27.771646 8043 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:27.734699 7933 consensus_queue.cc:1035] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.031s
I20260929 14:12:27.772782 8072 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:27.773407 8079 log.cc:924] Time spent T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20260929 14:12:27.765189 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:27.774384 5899 raft_consensus.cc:2379] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 6: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:12:27.758021 7879 raft_consensus.cc:729] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:27.775605 7879 consensus_queue.cc:260] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.776067 5351 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:27.777673 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1648) took 1121 ms. Trace:
I20260929 14:12:27.777840 8019 raft_consensus.cc:2726] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260929 14:12:27.778159 5193 rpcz_store.cc:276] 0929 14:12:26.656481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:26.757326 (+100845us) service_pool.cc:227] Handling call
0929 14:12:27.074196 (+316870us) spinlock_profiling.cc:243] Waited 7.01 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:12:27.137059 (+ 62863us) spinlock_profiling.cc:243] Waited 11.3 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0db7c 00007fbda1b0d73b 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:12:27.137256 (+ 197us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:27.158420 (+ 21164us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:27.158458 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:27.171286 (+ 12828us) log.cc:842] Serialized 26 byte log entry
0929 14:12:27.282714 (+111428us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:27.389553 (+106839us) log.cc:842] Serialized 14 byte log entry
0929 14:12:27.392908 (+ 3355us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:27.393015 (+ 107us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:27.777454 (+384439us) raft_consensus.cc:1671] finished
0929 14:12:27.777484 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:27.777614 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6028,"prepare.queue_time_us":49933,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":1327,"raft.queue_time_us":8829,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":16394,"spinlock_wait_cycles":42084992,"thread_start_us":172965,"threads_started":2,"wal-append.queue_time_us":240950}
I20260929 14:12:27.758229 8038 raft_consensus.cc:3037] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:27.764192 8021 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:27.727066 7992 consensus_queue.cc:1046] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260929 14:12:27.759536 7996 raft_consensus.cc:2726] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:27.796748 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:27.797389 8061 consensus_queue.cc:1035] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260929 14:12:27.737166 7979 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:27.807955 7979 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:27.809840 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:27.810154 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:27.811218 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:27.808678 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:27.815433 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 50365.7us
I20260929 14:12:27.817211 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:27.818220 5276 raft_consensus.cc:2379] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 7: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:27.808454 8038 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:27.796533 8021 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:27.821157 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:27.821902 5275 raft_consensus.cc:2379] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 5: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:27.832541 8060 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.832818 7869 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.841079 7311 raft_consensus.cc:3032] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Stepping down as leader of term 7
I20260929 14:12:27.841675 7311 raft_consensus.cc:729] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:27.842670 7311 consensus_queue.cc:260] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.844038 7311 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:27.848973 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:27.852680 5349 raft_consensus.cc:1226] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:27.850142 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:27.852370 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 215217us
W20260929 14:12:27.871711 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 119806us
I20260929 14:12:27.884236 5274 raft_consensus.cc:1226] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:27.889472 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:27.889763 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:27.892958 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:27.902895 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
W20260929 14:12:27.903930 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 258311us
I20260929 14:12:27.905875 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:27.911085 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:27.920030 7822 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.922091 8038 leader_election.cc:288] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:27.922523 7742 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:27.847465 8021 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:27.924849 8021 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:27.925662 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:27.926419 5899 raft_consensus.cc:2379] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 6: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:12:27.929371 5351 raft_consensus.cc:1226] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:27.934013 5197 raft_consensus.cc:1226] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:27.934345 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:27.940112 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:27.940696 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:27.949833 5193 raft_consensus.cc:1226] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:27.950281 7822 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:27.950966 7869 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:27.953689 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:27.954187 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:27.954425 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:27.956655 5165 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.960170 5165 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:27.960425 5239 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:27.934765 8023 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:27.964596 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:27.960458 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e4a096fb48c46dd95a1d74d34b6e670" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:27.965967 5267 raft_consensus.cc:1261] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:12:27.962248 5195 raft_consensus.cc:2379] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 6: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:27.943193 7765 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:27.975330 7982 raft_consensus.cc:2726] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:27.975397 7765 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:27.975589 5165 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.970057 8045 raft_consensus.cc:2726] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:27.977125 8045 raft_consensus.cc:2726] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:27.977566 7765 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:27.978418 5165 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:27.972937 7933 raft_consensus.cc:2781] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election won for term 8
W20260929 14:12:27.983310 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 76952.9us
W20260929 14:12:27.983690 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 53683.8us
I20260929 14:12:27.988588 5274 raft_consensus.cc:1261] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:27.988477 5165 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.993692 7765 raft_consensus.cc:2726] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260929 14:12:27.993896 5165 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.994726 7765 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:12:27.995000 5165 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:27.995561 5347 raft_consensus.cc:2454] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:27.995831 7765 raft_consensus.cc:2726] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:27.996161 5165 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:27.999708 5165 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:28.001664 8038 raft_consensus.cc:2726] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:12:28.004249 7682 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:28.005021 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:28.005317 8085 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:28.005772 8085 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.006641 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:28.007721 8085 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:28.009840 5870 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:28.010938 8077 raft_consensus.cc:2726] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:27.985105 8023 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.013942 8059 raft_consensus.cc:3032] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:28.014348 8059 raft_consensus.cc:729] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:28.014897 8045 raft_consensus.cc:2726] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:12:28.015134 8059 consensus_queue.cc:260] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.016201 8059 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:28.016634 5164 leader_election.cc:302] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:12:28.018256 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:28.019357 5870 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:28.019418 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:28.019847 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:28.020270 5267 raft_consensus.cc:2379] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 6: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:28.020113 5165 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:28.020539 5195 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:28.021958 8038 consensus_queue.cc:1046] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260929 14:12:28.024406 7917 raft_consensus.cc:2781] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:28.025288 8045 consensus_queue.cc:1046] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:12:28.026180 7982 raft_consensus.cc:3032] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:28.026520 7982 raft_consensus.cc:729] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 4 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:28.027431 7982 consensus_queue.cc:260] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.028218 5345 raft_consensus.cc:1261] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:28.027290 7903 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:28.028676 7982 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:28.029973 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:28.030622 5342 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 8.
I20260929 14:12:27.996717 7933 raft_consensus.cc:686] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:28.032245 7977 raft_consensus.cc:979] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:28.031899 7933 consensus_queue.cc:237] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.033687 7879 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:28.035373 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:28.035686 8023 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:28.036062 7977 raft_consensus.cc:1067] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:27.986187 7981 raft_consensus.cc:2726] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:28.037156 8060 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:27.997004 7649 consensus_queue.cc:1046] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260929 14:12:28.038560 7649 raft_consensus.cc:3032] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:28.038897 7649 raft_consensus.cc:729] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:28.038942 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:28.042645 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:28.042833 5196 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:12:28.045080 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:28.046711 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:28.047247 5272 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:28.043275 5350 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 8.
I20260929 14:12:28.053926 7917 raft_consensus.cc:686] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:28.043304 5196 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:28.054523 8038 raft_consensus.cc:3032] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:28.054697 5196 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:28.054955 8038 raft_consensus.cc:729] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 5 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:28.054821 7917 consensus_queue.cc:237] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:28.055858 8038 consensus_queue.cc:260] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.057039 8038 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:28.058182 7981 raft_consensus.cc:2726] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260929 14:12:28.061177 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 60322.9us
I20260929 14:12:28.062225 5165 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:28.063273 8039 raft_consensus.cc:2726] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260929 14:12:28.063300 5165 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:28.064445 5165 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:28.065500 7742 raft_consensus.cc:2726] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260929 14:12:28.066071 5165 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:28.066303 7742 raft_consensus.cc:2726] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:28.067463 5164 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:28.067570 7742 consensus_queue.cc:1035] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:28.068209 7765 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:28.068590 7765 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:28.068902 7765 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:28.074088 5164 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:28.039739 7649 consensus_queue.cc:260] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.077610 5272 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:28.077994 5196 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.094429 5195 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:28.098836 5316 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:28.066865 8045 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260929 14:12:28.113878 5196 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:28.114800 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:28.115455 5350 raft_consensus.cc:3032] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:28.115947 5350 raft_consensus.cc:729] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:28.066597 7841 consensus_queue.cc:260] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.116670 5350 consensus_queue.cc:260] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.117722 7841 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:28.118206 5350 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:28.119009 5314 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:28.120438 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:28.121054 5267 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:28.116678 7977 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:28.121667 7977 raft_consensus.cc:686] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:28.122591 7992 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:28.123364 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:28.120903 5314 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:28.122460 7977 consensus_queue.cc:237] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.092967 7981 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260929 14:12:28.125458 5350 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:28.126873 5164 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:28.128393 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:28.128976 5195 raft_consensus.cc:3037] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:28.131206 8097 raft_consensus.cc:2781] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:28.133064 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:28.140503 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:28.141111 5276 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 9.
I20260929 14:12:28.142233 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:28.142797 5195 raft_consensus.cc:2454] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:28.142846 5270 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 8.
I20260929 14:12:28.146325 5314 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:28.127238 7981 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:28.151587 7981 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:28.132344 7869 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260929 14:12:28.144079 7649 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:28.110586 7742 consensus_queue.cc:1035] T 4fc85ccb35394b2d800cea4f13552756 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.035s
I20260929 14:12:28.112524 8039 consensus_queue.cc:1035] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.026s
I20260929 14:12:28.158668 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:28.159294 5275 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:28.173444 7765 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.194437 8072 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:28.218284 5242 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:28.219901 8068 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260929 14:12:28.221385 7765 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:28.213047 8097 raft_consensus.cc:686] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:28.230255 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:28.230897 5276 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:28.233537 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:28.234069 5274 raft_consensus.cc:1261] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260929 14:12:28.234133 5350 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:28.220975 7981 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.253245 5276 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 9.
I20260929 14:12:28.256323 5165 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:28.261125 5350 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 9.
I20260929 14:12:28.263239 5274 raft_consensus.cc:1226] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 2. Current term is 3. Ops: []
W20260929 14:12:28.262431 8094 log.cc:924] Time spent T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e: Append to log took a long time: real 0.121s user 0.000s sys 0.001s
I20260929 14:12:28.269568 5096 catalog_manager.cc:5495] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:28.285756 5342 raft_consensus.cc:1261] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260929 14:12:28.258778 7960 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
I20260929 14:12:28.269901 8073 consensus_queue.cc:1046] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260929 14:12:28.294168 7992 raft_consensus.cc:2726] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:28.295575 7981 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 10 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:28.269320 7917 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election won for term 9
I20260929 14:12:28.296423 7917 raft_consensus.cc:686] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:28.299973 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:28.300647 5267 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:28.297564 8072 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:28.303109 8072 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:28.300585 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:28.305665 5342 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 10
I20260929 14:12:28.307945 5275 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:28.365749 5267 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 10.
W20260929 14:12:28.371868 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 73712.7us
I20260929 14:12:28.248745 8097 consensus_queue.cc:237] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.377525 5165 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:28.378666 7673 consensus_queue.cc:1035] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 14:12:28.379499 7742 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader election won for term 10
I20260929 14:12:28.379997 7742 raft_consensus.cc:686] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 10 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:28.297248 7917 consensus_queue.cc:237] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.380864 7742 consensus_queue.cc:237] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.390064 5096 catalog_manager.cc:5495] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 4 to 9, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 9 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:28.415257 5342 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 10.
I20260929 14:12:28.418486 5096 catalog_manager.cc:5495] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:28.421847 5096 catalog_manager.cc:5495] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:28.425421 5096 catalog_manager.cc:5495] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:28.426920 5096 catalog_manager.cc:5495] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:28.433091 5096 catalog_manager.cc:5495] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 6 to 7, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 7 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:28.440555 5200 raft_consensus.cc:1261] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260929 14:12:28.440976 5096 catalog_manager.cc:5495] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 2 to 5, leader changed from 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 5 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260929 14:12:28.443181 5096 catalog_manager.cc:5495] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 5 to 6, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 6 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:28.444763 5096 catalog_manager.cc:5495] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 4 to 5, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 5 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:28.446314 5192 raft_consensus.cc:2396] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260929 14:12:28.446942 5096 catalog_manager.cc:5495] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 5 to 10. New cstate: current_term: 10 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:28.448318 5096 catalog_manager.cc:5495] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:28.451508 5096 catalog_manager.cc:5495] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 4 to 5, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 5 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
W20260929 14:12:28.456760 5099 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52736 (request call id 224) took 12807 ms (client timeout 15000 ms). Trace:
I20260929 14:12:28.454176 5096 catalog_manager.cc:5495] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:28.460695 5096 catalog_manager.cc:5495] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 2 to 5, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 5 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:28.464588 5096 catalog_manager.cc:5495] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:28.485075 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:28.455530 8021 consensus_queue.cc:1035] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260929 14:12:28.484329 5096 catalog_manager.cc:5495] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 4 to 5, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 5 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
W20260929 14:12:28.457237 5099 rpcz_store.cc:269] 0929 14:12:15.648972 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:15.649206 (+ 234us) service_pool.cc:227] Handling call
0929 14:12:28.456671 (+12807465us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:12:26.001703 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:12:26.043867 (+ 42164us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:12:26.043928 (+ 61us) write_op.cc:435] Acquired schema lock
0929 14:12:26.043948 (+ 20us) tablet.cc:660] Decoding operations
0929 14:12:26.044739 (+ 791us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:12:26.044840 (+ 101us) write_op.cc:641] Partition lock acquired for write op
0929 14:12:26.044868 (+ 28us) tablet.cc:683] Acquiring locks for 6 operations
0929 14:12:26.045557 (+ 689us) tablet.cc:699] Row locks acquired
0929 14:12:26.045609 (+ 52us) write_op.cc:260] PREPARE: finished
0929 14:12:26.045813 (+ 204us) write_op.cc:270] Start()
0929 14:12:26.046000 (+ 187us) write_op.cc:276] Timestamp: P: 1790691146045741 usec, L: 0
0929 14:12:26.046046 (+ 46us) op_driver.cc:331] REPLICATION: starting
0929 14:12:26.046865 (+ 819us) log.cc:842] Serialized 2449 byte log entry
0929 14:12:27.157781 (+1110916us) op_driver.cc:444] REPLICATION: finished
0929 14:12:27.451001 (+293220us) write_op.cc:301] APPLY: starting
0929 14:12:27.463026 (+ 12025us) tablet.cc:1365] starting BulkCheckPresence
0929 14:12:27.496450 (+ 33424us) tablet.cc:1368] finished BulkCheckPresence
0929 14:12:27.496632 (+ 182us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:12:27.546140 (+ 49508us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:12:27.551195 (+ 5055us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=6
0929 14:12:27.551249 (+ 54us) write_op.cc:312] APPLY: finished
0929 14:12:27.551605 (+ 356us) log.cc:842] Serialized 53 byte log entry
0929 14:12:27.551765 (+ 160us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:27.551870 (+ 105us) write_op.cc:454] Released schema lock
0929 14:12:27.551915 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0929 14:12:27.552204 (+ 289us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:27.552237 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2238,"reported_tablets":37,"child_traces":[["op",{"apply.queue_time_us":257331,"apply.run_cpu_time_us":3888,"apply.run_wall_time_us":119835,"num_ops":6,"prepare.queue_time_us":266347,"prepare.run_cpu_time_us":18854,"prepare.run_wall_time_us":179912,"raft.queue_time_us":514793,"raft.run_cpu_time_us":6714,"raft.run_wall_time_us":141116,"replication_time_us":1104055,"thread_start_us":823637,"threads_started":4,"wal-append.queue_time_us":288115}]]}
I20260929 14:12:28.524501 5096 catalog_manager.cc:5495] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
W20260929 14:12:28.530803 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 88633.5us
I20260929 14:12:28.528321 5096 catalog_manager.cc:5495] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:28.532922 5096 catalog_manager.cc:5495] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:28.534914 5096 catalog_manager.cc:5495] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:28.539423 5096 catalog_manager.cc:5495] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 1 to 3, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 3 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:28.540890 5096 catalog_manager.cc:5495] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:28.527729 8072 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.547667 5096 catalog_manager.cc:5495] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:28.564538 5096 catalog_manager.cc:5495] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 6 to 7, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 7 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:28.567576 5096 catalog_manager.cc:5495] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 5 to 7, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 7 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:28.570102 5096 catalog_manager.cc:5495] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:28.571736 5096 catalog_manager.cc:5495] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 5 to 6, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 6 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:28.599427 8043 raft_consensus.cc:3032] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Stepping down as leader of term 2
I20260929 14:12:28.599875 8043 raft_consensus.cc:729] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:28.597038 7981 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:28.602617 7981 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.604463 5316 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:28.605563 7992 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:12:28.607046 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:28.607679 5274 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:28.604521 7981 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:28.612550 5347 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:28.613593 5165 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:28.613809 5347 raft_consensus.cc:595] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:28.614439 5347 raft_consensus.cc:603] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260929 14:12:28.595110 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 63659us
W20260929 14:12:28.642393 8109 log.cc:924] Time spent T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.077s user 0.001s sys 0.002s
I20260929 14:12:28.648041 8072 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:28.650475 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:28.651207 5349 raft_consensus.cc:1261] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260929 14:12:28.652204 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:28.652817 5343 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:28.655974 8100 consensus_queue.cc:1035] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 14:12:28.656420 5899 raft_consensus.cc:1261] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260929 14:12:28.600703 8043 consensus_queue.cc:260] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:28.659516 8043 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:28.662461 5342 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:28.663033 5342 raft_consensus.cc:595] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:28.663331 5342 raft_consensus.cc:603] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:12:28.664731 7765 consensus_queue.cc:1035] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:28.666231 7765 consensus_queue.cc:1225] T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer 675b77a0a88a4bf08134281fc0cd949e log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260929 14:12:28.667469 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 103300us
I20260929 14:12:28.672152 5349 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "599240a72b454fca841864f021dbbc64"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:28.672694 5349 raft_consensus.cc:595] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:28.673674 5345 raft_consensus.cc:1261] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260929 14:12:28.675940 5345 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:28.676430 5345 raft_consensus.cc:595] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:28.688261 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:28.689946 5349 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:28.690495 5349 raft_consensus.cc:595] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:28.690804 5349 raft_consensus.cc:603] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:12:28.637701 7981 raft_consensus.cc:2781] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:28.693940 7682 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:28.694697 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260929 14:12:28.695101 5314 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:28.692921 5899 pending_rounds.cc:76] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e: Aborting all ops after (but not including) 5
I20260929 14:12:28.699811 5899 pending_rounds.cc:97] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260929 14:12:28.699908 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:12:28.700153 5899 raft_consensus.cc:2866] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:12:28.700637 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:28.712731 5193 raft_consensus.cc:1261] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260929 14:12:28.723737 7976 consensus_queue.cc:1035] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:12:28.702840 8009 raft_consensus.cc:2726] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260929 14:12:28.706727 7682 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.727488 7682 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:28.731155 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:28.731805 5198 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 3.
I20260929 14:12:28.712248 8021 consensus_queue.cc:1035] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260929 14:12:28.736333 5316 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:28.746171 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:28.697952 7981 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:28.748459 5274 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 2.
I20260929 14:12:28.748785 8057 raft_consensus.cc:2781] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:28.761488 8057 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:28.761808 8057 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:28.752285 7981 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:28.784425 5267 raft_consensus.cc:1261] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260929 14:12:28.794720 7992 consensus_queue.cc:1035] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260929 14:12:29.006976 5350 raft_consensus.cc:1261] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260929 14:12:29.014883 5274 raft_consensus.cc:1261] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260929 14:12:29.024011 5270 raft_consensus.cc:1261] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260929 14:12:29.026240 8045 consensus_queue.cc:1035] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:29.030553 8057 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:28.997200 7981 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:29.034832 7981 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:29.012359 7992 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:29.072831 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:29.076169 5276 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:29.083622 7981 consensus_queue.cc:1035] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:12:29.090443 5267 raft_consensus.cc:1261] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260929 14:12:29.085341 8068 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:29.095378 8068 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:29.097311 8070 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:29.097833 8070 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:29.099920 8070 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:29.103169 8100 consensus_queue.cc:1035] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:29.091820 7917 consensus_queue.cc:1035] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260929 14:12:29.069756 8072 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:29.123735 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:29.124759 5198 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 7.
I20260929 14:12:29.117120 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:29.092324 8086 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:29.135042 5272 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:29.092200 8061 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:29.141355 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:29.138223 5314 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:29.142076 5267 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 2 FOLLOWER]: Advancing to term 4
I20260929 14:12:29.141817 5242 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
W20260929 14:12:29.133369 8099 log.cc:924] Time spent T 0e4a096fb48c46dd95a1d74d34b6e670 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260929 14:12:29.147522 5276 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:29.077118 7992 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:29.133527 8057 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:29.159730 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 189850us
I20260929 14:12:29.156507 5165 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:29.163350 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:29.164158 5198 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:29.167443 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:29.168133 5200 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:29.150192 8068 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:29.153445 7682 raft_consensus.cc:2781] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:29.154786 7996 raft_consensus.cc:2781] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:29.170295 5267 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:29.135162 8086 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:29.129835 8072 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:29.166397 8097 raft_consensus.cc:2781] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:29.198626 8072 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:29.199687 8097 raft_consensus.cc:686] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:29.200659 8097 consensus_queue.cc:237] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:29.201387 7992 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:29.150509 8061 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:29.179272 7996 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:29.179356 7682 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:29.185994 8068 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:29.220780 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:29.221463 5196 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
W20260929 14:12:29.246729 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 223617us
I20260929 14:12:29.251533 5342 raft_consensus.cc:1261] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260929 14:12:29.254388 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:29.232671 5198 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:29.254972 5342 raft_consensus.cc:3037] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:29.220726 7996 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:29.260730 5314 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:29.261334 5350 raft_consensus.cc:1226] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:29.258929 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:29.264212 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:29.262444 5193 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:29.220916 7682 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:29.270479 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:29.264911 5274 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 4.
I20260929 14:12:29.272363 5342 raft_consensus.cc:2454] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:29.281363 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:29.282025 5274 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:29.241297 8068 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:29.330103 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:29.331032 5198 raft_consensus.cc:2379] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 6: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:29.331431 5342 raft_consensus.cc:1261] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260929 14:12:29.294533 8124 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:29.362761 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 102096us
I20260929 14:12:29.330086 8072 raft_consensus.cc:2781] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:12:29.388482 5314 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:29.396108 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 142536us
I20260929 14:12:29.418747 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:29.419687 5342 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:29.421782 8072 raft_consensus.cc:686] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:29.427986 5239 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:29.409474 8061 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:29.387336 7649 consensus_queue.cc:1035] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260929 14:12:29.390642 7981 consensus_queue.cc:1046] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260929 14:12:29.426771 8086 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:29.403403 8100 consensus_queue.cc:1035] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260929 14:12:29.426558 8132 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:29.347087 8124 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:29.478586 8070 raft_consensus.cc:979] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:29.492949 7321 raft_consensus.cc:2781] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260929 14:12:29.480708 8126 log.cc:924] Time spent T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.174s user 0.000s sys 0.000s
I20260929 14:12:29.493652 7321 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:29.494133 7321 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
W20260929 14:12:29.481350 8129 log.cc:924] Time spent T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.106s user 0.000s sys 0.001s
I20260929 14:12:29.481698 7933 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:29.487640 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 113030us
I20260929 14:12:29.448057 8125 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:29.495170 7933 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:29.456352 8116 raft_consensus.cc:2781] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260929 14:12:29.479581 8127 log.cc:924] Time spent T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29: Append to log took a long time: real 0.163s user 0.001s sys 0.001s
I20260929 14:12:29.495429 8125 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:29.504908 7933 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:29.511205 8128 raft_consensus.cc:3032] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Stepping down as leader of term 7
I20260929 14:12:29.493872 7996 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:29.422878 8072 consensus_queue.cc:237] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:29.512842 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:29.669992 5350 raft_consensus.cc:2379] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 5: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
W20260929 14:12:29.669116 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 280783us
W20260929 14:12:29.567365 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 168958us
I20260929 14:12:29.671320 5314 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:29.546294 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:29.674588 5198 raft_consensus.cc:2396] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260929 14:12:29.520625 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:29.680675 5193 raft_consensus.cc:2379] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 8: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
W20260929 14:12:29.681460 8134 log.cc:924] Time spent T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.269s user 0.003s sys 0.000s
I20260929 14:12:29.542958 8051 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:29.669770 8128 raft_consensus.cc:729] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:29.653654 8115 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:29.587399 8021 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:29.501715 8116 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:29.624920 8140 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:29.703001 7976 raft_consensus.cc:979] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:29.697686 7742 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:29.705513 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:29.680653 8068 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:29.497864 8132 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:29.710742 5242 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:29.967876 8141 log.cc:924] Time spent T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.315s user 0.000s sys 0.001s
I20260929 14:12:30.116477 7406 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:29.821807 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260929 14:12:30.117852 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 407147us
W20260929 14:12:29.716777 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 45541.8us
I20260929 14:12:29.756153 8148 raft_consensus.cc:2781] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260929 14:12:29.846160 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 269461us
W20260929 14:12:30.114773 7321 consensus_meta.cc:287] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.620s user 0.011s sys 0.000s
I20260929 14:12:30.131114 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:30.173475 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:30.174363 5274 raft_consensus.cc:2379] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 3: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:30.175048 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:30.177963 5270 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:30.160703 5267 raft_consensus.cc:1261] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260929 14:12:30.186625 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:30.187279 5196 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
W20260929 14:12:29.991727 8147 log.cc:924] Time spent T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.136s user 0.000s sys 0.000s
I20260929 14:12:29.818687 8149 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:30.063683 7682 consensus_meta.cc:287] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.777s user 0.013s sys 0.001s
I20260929 14:12:30.190431 5349 raft_consensus.cc:1226] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:29.833612 8124 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.173643 8158 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:30.174401 8009 raft_consensus.cc:1067] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:30.214998 5345 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:29.574674 7879 raft_consensus.cc:1067] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:30.226733 5345 raft_consensus.cc:595] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:30.189157 7321 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.194418 7649 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:30.125051 8116 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:30.128544 8138 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:30.229444 5349 raft_consensus.cc:1261] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260929 14:12:30.128798 8057 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:30.075335 8125 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:30.132820 7996 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.125934 8148 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:30.237375 8148 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:30.237562 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:30.238236 5906 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 3.
I20260929 14:12:30.238613 8138 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.240204 5346 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:30.240648 8138 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:30.212924 5196 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:30.171660 7742 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.241818 5200 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:30.241804 8057 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.242532 7742 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.243983 8057 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:30.244220 8132 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.245851 7321 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:30.257357 8152 consensus_queue.cc:1035] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:12:30.257539 7649 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.207144 8146 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:30.259594 7649 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.259344 8146 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:30.225764 8158 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.262146 8158 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.175141 8140 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.264781 8140 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:30.226577 8128 consensus_queue.cc:260] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.266242 8128 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:30.269728 5346 raft_consensus.cc:595] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:30.270159 5346 raft_consensus.cc:603] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:12:29.718253 8051 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.218425 7682 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.290634 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:30.277869 8146 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:30.280859 7406 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:30.207681 8149 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.283067 5167 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:30.273303 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:30.285763 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:30.290112 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:30.296548 5351 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 8.
I20260929 14:12:30.205264 8045 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:30.129561 8021 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.291340 5343 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
W20260929 14:12:30.276656 8153 consensus_queue.cc:1159] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" responder_term: 7 status { last_received { term: 7 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 7 index: 5 } }
I20260929 14:12:30.175680 8115 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.296425 5344 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:30.300230 8045 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.295882 5349 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:30.305685 8149 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:30.306166 5347 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:30.307139 7649 consensus_queue.cc:1035] T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:12:30.295742 8132 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:30.310528 8132 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.312343 8132 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.319861 5164 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:30.329521 5164 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:30.326663 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 147532us
I20260929 14:12:30.330528 5164 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:30.336781 5164 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:30.362659 5316 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:30.364637 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 149254us
W20260929 14:12:30.367304 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 141121us
I20260929 14:12:30.368803 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:30.372043 8125 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:30.372596 8125 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.343102 7765 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:30.373431 7765 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:30.373793 7765 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:30.374665 8125 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.376804 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:30.378425 8051 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.330324 8132 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:30.379199 8132 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.326115 8155 raft_consensus.cc:2781] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 14:12:30.386390 8155 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:30.386945 8155 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 2 FOLLOWER]: Advancing to term 3
I20260929 14:12:30.388949 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:30.361720 8162 raft_consensus.cc:2781] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:30.389739 5343 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 8
I20260929 14:12:30.394423 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:30.395205 5351 raft_consensus.cc:2379] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 6: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:30.396211 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:30.397708 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:30.361371 8143 raft_consensus.cc:2781] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:30.399013 5348 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:30.401252 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:30.414549 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 51975.4us
I20260929 14:12:30.362210 8038 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:30.429193 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:30.431016 5199 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "599240a72b454fca841864f021dbbc64"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:12:30.432049 5199 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:30.422267 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:30.421267 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:30.422005 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:30.422384 5193 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:12:30.423269 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:30.432078 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 304437us
I20260929 14:12:30.450193 5199 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:30.451892 5267 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:30.430918 5270 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:30.433305 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:30.443817 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:30.450554 5239 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:30.450886 5198 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:30.450213 5269 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:30.432163 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:30.433209 8135 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:30.435513 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:30.466094 5276 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:30.451167 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:30.449920 5193 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:30.453440 5906 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:12:30.453891 5196 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:30.394047 8162 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:30.467741 8162 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:30.465665 5242 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:30.467159 5343 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:30.473423 8157 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:30.474004 8157 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.474437 5196 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:30.474803 5242 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:30.474968 8021 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.475582 7682 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:30.475800 8061 raft_consensus.cc:2781] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:30.476138 8157 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.476305 8130 raft_consensus.cc:2726] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260929 14:12:30.476495 8061 raft_consensus.cc:686] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:30.477495 8061 consensus_queue.cc:237] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.479813 5242 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
W20260929 14:12:30.419001 8129 log.cc:924] Time spent T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260929 14:12:30.415814 8143 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:30.467185 5193 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:30.483966 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:30.467185 5268 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:30.487226 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:30.487449 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:30.548995 8115 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.550185 8162 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.555495 8162 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.551086 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:30.547426 7996 raft_consensus.cc:2781] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:30.556799 5165 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:30.568966 5192 raft_consensus.cc:2354] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:30.559917 5193 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:30.569494 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 81094.9us
I20260929 14:12:30.429039 8038 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:30.559372 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:30.571461 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:30.571784 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:30.561092 5343 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:30.572713 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:30.573562 5276 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 7
I20260929 14:12:30.573990 5239 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:30.572651 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:30.575819 5198 raft_consensus.cc:2379] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 5: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:30.568629 7996 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:30.572824 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:30.577042 7996 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:30.577485 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:30.491865 8148 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.465768 8135 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:30.581808 8135 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:30.581350 8100 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:30.585391 8100 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:30.585800 8100 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:30.481294 8146 raft_consensus.cc:2781] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:30.587445 8146 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:30.587802 8146 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:30.556453 8143 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:30.591763 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:30.592312 5200 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:30.591432 7996 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:30.593590 5276 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:30.596691 7996 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:30.591449 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:30.598935 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:30.599493 5276 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:30.604517 5200 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:30.606523 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:30.615250 5193 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.615592 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:30.617368 5267 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 6
I20260929 14:12:30.619592 5242 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:30.622105 5165 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:30.625670 5270 raft_consensus.cc:2379] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 7: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
W20260929 14:12:30.635075 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 141980us
I20260929 14:12:30.636068 5276 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:30.646927 5267 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:30.653065 8045 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.653862 8148 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:30.654101 8132 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.653903 8131 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:30.563282 8116 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.655095 8068 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:30.655144 8131 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.655671 8068 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:30.657094 8155 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:30.657640 8068 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:30.658413 8131 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.659216 8155 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:30.598217 8038 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:30.669390 5195 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:12:30.701804 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:30.733340 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 111334us
I20260929 14:12:30.735252 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:30.735455 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:30.736261 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 59772us
I20260929 14:12:30.737577 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:30.742909 5344 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:30.743458 5344 raft_consensus.cc:595] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
W20260929 14:12:30.746022 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 155405us
I20260929 14:12:30.749177 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:30.749836 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:30.750018 5193 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:30.750602 5276 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:30.746439 5199 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.744836 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:30.779805 5899 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:30.783344 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:30.792050 5199 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.793545 5269 raft_consensus.cc:2354] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:12:30.807708 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:30.807578 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:30.727018 7996 raft_consensus.cc:2726] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:30.811712 5348 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:30.748680 7981 raft_consensus.cc:2781] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:30.823436 7981 raft_consensus.cc:686] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:30.824342 7981 consensus_queue.cc:237] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.813210 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:30.826802 5344 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:30.813072 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:30.829630 5345 raft_consensus.cc:3032] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:30.829954 5345 raft_consensus.cc:729] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:30.830664 5345 consensus_queue.cc:260] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.831614 5345 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:30.822644 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:30.833251 5347 raft_consensus.cc:3032] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Stepping down as leader of term 8
I20260929 14:12:30.833542 5347 raft_consensus.cc:729] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:30.816284 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:30.835847 5350 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:30.836289 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:30.837658 5164 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:30.834896 5239 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:30.840284 5899 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:30.834196 5347 consensus_queue.cc:260] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.845233 5347 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:30.845316 5348 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:30.848351 5193 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:30.836707 5350 raft_consensus.cc:595] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:30.855975 5347 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 9.
I20260929 14:12:30.862635 5164 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:30.748562 7765 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.844478 8125 raft_consensus.cc:2781] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:30.869065 8128 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election won for term 9
I20260929 14:12:30.875845 5347 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:30.876778 5347 raft_consensus.cc:595] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:30.881918 5196 raft_consensus.cc:2379] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 9: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:30.883014 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260929 14:12:30.881465 8128 raft_consensus.cc:686] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:30.871064 8131 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:30.886029 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:12:30.906984 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:30.870772 8125 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:30.909976 8128 consensus_queue.cc:237] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.906050 8131 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:30.923365 8125 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:30.925303 8168 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:30.952484 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 180741us
I20260929 14:12:30.940955 8131 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:30.969712 5242 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:30.972185 5316 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:30.951118 8116 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:30.946298 8135 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.941547 8100 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:30.975437 8130 raft_consensus.cc:2781] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 14:12:30.950485 8168 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.020069 5345 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:31.021436 5201 raft_consensus.cc:2379] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 9: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:31.022264 8100 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:31.023223 8181 log.cc:924] Time spent T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.090s user 0.000s sys 0.000s
W20260929 14:12:31.024622 8178 log.cc:924] Time spent T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.100s user 0.000s sys 0.001s
I20260929 14:12:31.025707 8130 raft_consensus.cc:686] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:31.026660 8130 consensus_queue.cc:237] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:31.033716 8038 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.034997 5197 raft_consensus.cc:2354] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:31.035677 8038 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:31.037362 8135 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:31.038165 5342 raft_consensus.cc:2354] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:31.022873 8143 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.020109 8068 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.055992 5267 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:31.056967 8068 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:31.081569 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 90285.5us
W20260929 14:12:31.116454 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 281607us
W20260929 14:12:31.121290 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 128513us
I20260929 14:12:31.123643 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:31.080150 7765 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:31.129334 5349 raft_consensus.cc:2354] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:31.138134 8125 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.142495 8190 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:31.143071 8190 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.145723 8190 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:31.148882 8068 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:31.149652 8068 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.150257 8143 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:31.058341 8146 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.151835 8068 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:31.152547 5268 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:31.153965 8068 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:31.155192 8068 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:31.157610 8068 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:31.162633 5239 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:31.165241 5239 leader_election.cc:398] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:31.165704 5239 leader_election.cc:401] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260929 14:12:31.163194 8125 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:31.166638 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:31.167205 8068 raft_consensus.cc:2726] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:31.167912 8021 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:31.176786 5345 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:31.152946 8146 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:31.148840 8168 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:31.190801 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:31.190071 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:31.189339 5239 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:31.193032 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:31.194005 5347 raft_consensus.cc:2379] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 9: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:31.195571 5345 raft_consensus.cc:1261] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260929 14:12:31.140805 8131 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.204276 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:31.204202 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:31.204218 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:31.210103 8143 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:31.210592 8143 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.212507 8143 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:31.214395 5345 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:31.214546 8143 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:31.215061 8143 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.203215 8190 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:31.216607 5165 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:31.218036 8154 raft_consensus.cc:2781] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:31.218587 8143 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:31.222311 5165 leader_election.cc:398] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:12:31.224936 5165 leader_election.cc:401] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260929 14:12:31.199760 8125 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:31.224960 7765 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:31.227613 8143 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:31.228906 7765 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.231019 7765 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:31.231256 5242 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:31.235008 8061 consensus_queue.cc:1035] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:31.225419 8154 raft_consensus.cc:686] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:31.238489 7765 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:31.239893 8131 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:31.240002 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:31.241555 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:31.241648 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:31.280763 5268 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:31.280826 5271 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:31.279533 5242 leader_election.cc:398] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:31.238502 8154 consensus_queue.cc:237] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.244932 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:31.281991 5242 leader_election.cc:401] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:12:31.282253 5269 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:31.245971 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:31.247128 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:31.283349 5270 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:31.247820 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:31.284587 5350 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:31.247143 8148 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:31.226406 8190 raft_consensus.cc:686] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:31.251279 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:31.253022 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:31.311965 5346 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:31.254508 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:31.255174 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:31.255827 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 283698us
I20260929 14:12:31.256022 8135 raft_consensus.cc:2781] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:31.256789 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:31.257354 8125 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.260048 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:31.320817 5275 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:31.261752 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:31.321141 5239 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:31.270287 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:31.322463 5272 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 7
I20260929 14:12:31.271832 5314 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:31.325130 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 53400.7us
I20260929 14:12:31.326052 5314 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:31.277948 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:31.327523 5276 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 8
I20260929 14:12:31.242199 7321 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:31.242259 8168 raft_consensus.cc:2726] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:31.300835 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:31.321615 5347 raft_consensus.cc:2379] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 7: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:31.262408 5349 raft_consensus.cc:1261] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:31.334497 5350 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 9.
I20260929 14:12:31.338900 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:31.345621 5268 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 9.
I20260929 14:12:31.348388 5274 raft_consensus.cc:1226] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:31.355195 5273 raft_consensus.cc:1261] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:12:31.355860 8140 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:31.361043 8021 raft_consensus.cc:2726] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:31.363817 7976 raft_consensus.cc:2781] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:31.367297 7976 raft_consensus.cc:686] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:31.368155 7976 consensus_queue.cc:237] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.365594 7321 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:31.372332 5269 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:31.374795 7321 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:31.319846 8135 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:31.376405 8135 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:31.379645 5346 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:31.320248 8148 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.381620 8148 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:31.317813 7765 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:31.475437 5271 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 9.
I20260929 14:12:31.480175 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:31.480921 5345 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:31.480470 8190 consensus_queue.cc:237] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.488344 5314 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:31.488548 5275 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:31.489593 8131 raft_consensus.cc:2741] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260929 14:12:31.494416 8135 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.496399 8135 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:31.496477 8166 raft_consensus.cc:979] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:31.496971 8166 raft_consensus.cc:1067] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:31.497623 5348 raft_consensus.cc:2379] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 6: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
W20260929 14:12:31.498040 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 218696us
I20260929 14:12:31.499217 5242 leader_election.cc:398] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:31.499485 5242 leader_election.cc:401] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:12:31.500017 8130 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:31.487267 5276 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:31.416986 5272 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:31.506896 5270 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:31.509135 8130 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:31.511312 7765 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:31.492017 8143 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 3. Current term is 4.
I20260929 14:12:31.513329 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:31.513581 5316 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:31.514904 8148 raft_consensus.cc:2781] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:31.515347 8148 raft_consensus.cc:686] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:31.516304 8166 raft_consensus.cc:979] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:31.516191 8148 consensus_queue.cc:237] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.518352 5314 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:31.526508 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:31.527464 5348 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:31.507993 5345 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 9.
I20260929 14:12:31.534920 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:31.535616 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:31.535688 5164 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:31.536019 5239 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:31.535610 5199 raft_consensus.cc:2354] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9.
I20260929 14:12:31.536758 5164 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:31.537667 5239 leader_election.cc:398] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:31.535717 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:31.539657 5194 raft_consensus.cc:2354] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 5. Current term is 6.
I20260929 14:12:31.532575 5316 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:31.534279 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:31.543273 5201 raft_consensus.cc:2354] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:31.543767 7765 raft_consensus.cc:2781] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election won for term 9
I20260929 14:12:31.544323 7765 raft_consensus.cc:686] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:31.545145 7765 consensus_queue.cc:237] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.547197 8038 raft_consensus.cc:2781] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:31.547659 8038 raft_consensus.cc:686] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:31.548506 8038 consensus_queue.cc:237] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.554136 8143 consensus_queue.cc:1035] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:31.557188 5899 raft_consensus.cc:1261] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:12:31.560238 8125 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:31.540062 8186 raft_consensus.cc:2741] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260929 14:12:31.541656 8189 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:31.530618 8137 raft_consensus.cc:2741] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260929 14:12:31.529209 8140 raft_consensus.cc:1067] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:31.566629 8135 raft_consensus.cc:2781] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:31.567253 8135 raft_consensus.cc:686] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:31.570849 8186 consensus_queue.cc:1035] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:12:31.577584 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:31.578454 5348 raft_consensus.cc:2354] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9.
I20260929 14:12:31.568365 8135 consensus_queue.cc:237] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.718284 7959 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:31.877148 7959 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:31.899029 7959 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:31.907802 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 337312us
W20260929 14:12:31.925591 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 360849us
I20260929 14:12:31.926685 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:31.932456 5097 catalog_manager.cc:5495] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 5 to 6, leader changed from 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 6 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
W20260929 14:12:31.937304 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 232630us
I20260929 14:12:31.945541 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:31.948222 5199 raft_consensus.cc:1261] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260929 14:12:31.948376 5196 raft_consensus.cc:2354] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9.
W20260929 14:12:31.950357 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 371820us
I20260929 14:12:31.939927 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:31.954766 5195 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:31.961198 8132 consensus_queue.cc:1035] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:31.966495 8132 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:31.928953 8051 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:31.969780 5242 leader_election.cc:398] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9.
I20260929 14:12:31.972658 5242 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:31.936713 8072 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:31.950850 8128 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:31.967140 8132 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.941018 8045 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:31.976733 8152 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s user 0.019s sys 0.012s
W20260929 14:12:31.990702 8152 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s user 0.020s sys 0.013s
I20260929 14:12:31.990958 8190 raft_consensus.cc:2726] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:31.991276 8152 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:31.991842 8152 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.981503 8072 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:31.995477 8152 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:31.996209 8072 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:31.970892 7996 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:31.989782 8130 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:31.989689 8045 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:31.969691 8115 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:31.976079 8051 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:31.985675 8128 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.013173 7996 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.014225 8130 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.022635 8115 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:32.113906 7976 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s user 0.018s sys 0.032s
I20260929 14:12:32.124493 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:32.125823 5201 raft_consensus.cc:2379] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 7: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:32.132671 5200 raft_consensus.cc:1226] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:32.139887 5193 raft_consensus.cc:1261] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260929 14:12:32.139561 5097 catalog_manager.cc:5495] T d4dd0a13d8aa4e6bbd3633c047984f67 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 4 to 7, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 7 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:32.141680 5097 catalog_manager.cc:5495] T 4300e83384e9458bb280a4b1e3131932 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 3 to 4, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 4 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:32.153321 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:32.153848 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:32.156100 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 226787us
I20260929 14:12:32.118026 8132 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:32.136164 7976 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.636s user 0.019s sys 0.032s
I20260929 14:12:32.165716 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:32.166339 5197 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:32.164099 8045 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:32.164988 8051 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:32.183113 5201 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:12:32.183812 5201 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:32.184193 5201 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:32.183041 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:32.180815 7996 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:32.189961 5197 raft_consensus.cc:2379] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 6: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:32.182090 8128 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:32.193842 5197 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:12:32.185526 8115 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:32.197463 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:32.198421 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:32.203178 5350 raft_consensus.cc:2379] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 8: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:32.199249 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:32.204260 5239 leader_election.cc:398] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9.
I20260929 14:12:32.197468 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:32.204708 5239 leader_election.cc:401] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260929 14:12:32.200307 8051 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:32.191184 8130 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:32.211869 8131 raft_consensus.cc:2726] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9.
I20260929 14:12:32.213320 8051 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:32.215296 8203 log.cc:924] Time spent T e50b67ada64545a6b7525d4aba317bb0 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260929 14:12:32.209442 5197 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:32.209741 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:32.221014 5197 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:32.222604 5097 catalog_manager.cc:5495] T e5acb0315ba44b5a9d145119868e7168 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 8 to 10, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68). New cstate: current_term: 10 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:32.223959 5316 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:12:32.227149 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 254510us
I20260929 14:12:32.228237 5242 leader_election.cc:398] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 5. Current term is 6.
I20260929 14:12:32.228462 5096 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52750 (request call id 223) took 6880 ms. Trace:
I20260929 14:12:32.234508 5165 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:32.215296 8051 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:32.236716 5165 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:32.238932 5165 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:32.241636 8190 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:32.241671 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:32.242323 8190 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.244287 8190 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:32.248790 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:32.250245 8100 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:32.250716 8100 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:32.251159 8100 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:32.253300 8132 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:32.253340 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:32.254473 5275 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:32.254124 8132 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.256611 8132 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:32.258955 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:32.259940 5275 raft_consensus.cc:2379] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 6: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:32.262215 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:32.229202 5242 leader_election.cc:398] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:32.271498 5165 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:32.284094 5276 raft_consensus.cc:1261] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:32.285614 5165 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:32.286900 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:32.290712 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:32.292446 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:32.295578 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:32.237051 8115 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:32.232064 5096 rpcz_store.cc:276] 0929 14:12:25.342543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:25.342835 (+ 292us) service_pool.cc:227] Handling call
0929 14:12:32.222263 (+6879428us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:12:28.972631 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:12:28.973104 (+ 473us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:12:28.973137 (+ 33us) write_op.cc:435] Acquired schema lock
0929 14:12:28.973153 (+ 16us) tablet.cc:660] Decoding operations
0929 14:12:28.974957 (+ 1804us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:12:28.975027 (+ 70us) write_op.cc:641] Partition lock acquired for write op
0929 14:12:28.975058 (+ 31us) tablet.cc:683] Acquiring locks for 28 operations
0929 14:12:28.990802 (+ 15744us) tablet.cc:699] Row locks acquired
0929 14:12:28.990821 (+ 19us) write_op.cc:260] PREPARE: finished
0929 14:12:28.990963 (+ 142us) write_op.cc:270] Start()
0929 14:12:28.991069 (+ 106us) write_op.cc:276] Timestamp: P: 1790691148990916 usec, L: 0
0929 14:12:28.991089 (+ 20us) op_driver.cc:331] REPLICATION: starting
0929 14:12:28.991669 (+ 580us) log.cc:842] Serialized 10951 byte log entry
0929 14:12:30.228049 (+1236380us) op_driver.cc:444] REPLICATION: finished
0929 14:12:30.832255 (+604206us) write_op.cc:301] APPLY: starting
0929 14:12:30.836296 (+ 4041us) tablet.cc:1365] starting BulkCheckPresence
0929 14:12:30.939598 (+103302us) tablet.cc:1368] finished BulkCheckPresence
0929 14:12:30.939784 (+ 186us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:12:31.145972 (+206188us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:12:31.146482 (+ 510us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=28
0929 14:12:31.146554 (+ 72us) write_op.cc:312] APPLY: finished
0929 14:12:31.147075 (+ 521us) log.cc:842] Serialized 188 byte log entry
0929 14:12:31.200359 (+ 53284us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:31.223016 (+ 22657us) write_op.cc:454] Released schema lock
0929 14:12:31.223108 (+ 92us) write_op.cc:341] FINISH: Updating metrics
0929 14:12:31.284767 (+ 61659us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:31.300372 (+ 15605us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":143770,"reported_tablets":41,"child_traces":[["op",{"apply.queue_time_us":509154,"apply.run_cpu_time_us":20225,"apply.run_wall_time_us":575567,"num_ops":28,"prepare.queue_time_us":264479,"prepare.run_cpu_time_us":81567,"prepare.run_wall_time_us":247624,"raft.queue_time_us":553041,"raft.run_cpu_time_us":20926,"raft.run_wall_time_us":257292,"replication_time_us":1224359,"thread_start_us":603197,"threads_started":5,"wal-append.queue_time_us":807097,"wal-append.run_cpu_time_us":14115,"wal-append.run_wall_time_us":1511975}]]}
I20260929 14:12:32.304301 5197 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.277925 8154 raft_consensus.cc:2726] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:32.313862 5343 raft_consensus.cc:1226] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:32.314047 5199 raft_consensus.cc:1226] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:32.316512 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:32.284814 8045 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election won for term 9
I20260929 14:12:32.236893 7321 consensus_queue.cc:1035] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
I20260929 14:12:32.318145 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:32.317951 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:32.309998 8132 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:32.319205 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 367107us
I20260929 14:12:32.319587 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:32.332285 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:32.333387 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:32.320008 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:32.333884 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:32.334255 5194 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:32.334460 5273 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:32.335034 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:32.320281 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:32.291989 8143 raft_consensus.cc:2781] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:32.336532 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:32.320740 5192 raft_consensus.cc:1261] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260929 14:12:32.320843 5197 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:32.323910 5242 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:32.305601 8072 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260929 14:12:32.390537 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 66862.2us
W20260929 14:12:32.390621 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 48592.2us
I20260929 14:12:32.312330 8155 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election won for term 7
W20260929 14:12:32.398608 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 76229us
I20260929 14:12:32.403255 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:32.404335 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:32.404827 5350 raft_consensus.cc:3032] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:32.405548 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:32.410595 5344 raft_consensus.cc:1261] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260929 14:12:32.388900 5201 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.411450 5193 raft_consensus.cc:1226] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:32.400800 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:32.402545 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:32.412923 5347 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 8.
I20260929 14:12:32.414250 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:32.414858 5200 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:32.405092 5350 raft_consensus.cc:729] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:32.416975 5201 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:32.419427 5351 raft_consensus.cc:1261] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:32.422354 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:32.425491 5242 leader_election.cc:398] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9.
I20260929 14:12:32.419454 5350 consensus_queue.cc:260] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.426605 5242 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:32.426952 5350 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:32.314260 8115 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.398159 8045 raft_consensus.cc:686] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:32.419642 8155 raft_consensus.cc:686] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
W20260929 14:12:32.458374 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 234490us
I20260929 14:12:32.423300 8214 raft_consensus.cc:2726] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:32.443192 5350 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:32.418148 8038 consensus_queue.cc:1035] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260929 14:12:32.464583 5316 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:32.444833 8190 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:32.499970 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:32.392156 8132 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.514204 5899 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 10.
I20260929 14:12:32.430414 8189 consensus_queue.cc:1046] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
W20260929 14:12:32.539956 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 75425.5us
I20260929 14:12:32.532271 8143 raft_consensus.cc:686] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:32.548246 5270 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:12:32.511332 8154 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:32.522095 8190 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:32.559108 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 47482.7us
I20260929 14:12:32.447818 8212 consensus_queue.cc:1035] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.116s
I20260929 14:12:32.567557 5167 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
W20260929 14:12:32.560961 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 169202us
I20260929 14:12:32.583105 5165 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:32.590705 5199 raft_consensus.cc:1261] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260929 14:12:32.557767 7321 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:32.563146 8190 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:32.612773 8234 consensus_queue.cc:1046] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:12:32.616830 5350 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:32.617669 5350 raft_consensus.cc:595] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:32.618067 5350 raft_consensus.cc:603] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:12:32.621721 5344 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:32.622284 5344 raft_consensus.cc:595] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:32.622610 5344 raft_consensus.cc:603] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:12:32.599700 7321 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:32.564234 8154 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.633876 8038 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260929 14:12:32.647261 5200 raft_consensus.cc:1261] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:32.651803 5199 raft_consensus.cc:1261] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260929 14:12:32.646051 8038 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:32.653564 8038 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Advancing to term 11
I20260929 14:12:32.644619 8208 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:32.651829 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:32.655023 5342 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:32.660534 5342 raft_consensus.cc:595] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:12:32.661690 8216 log.cc:924] Time spent T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.071s user 0.000s sys 0.000s
W20260929 14:12:32.662956 8217 log.cc:924] Time spent T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
I20260929 14:12:32.663622 5164 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:32.664944 8233 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:32.570277 8155 consensus_queue.cc:237] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.665380 8233 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:32.665773 8233 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:32.666066 8208 consensus_queue.cc:1035] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:32.671154 7976 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:32.671463 8115 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:32.671859 7976 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.672657 8132 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:32.673285 8232 consensus_queue.cc:1035] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:32.675074 8232 consensus_queue.cc:1035] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:32.676052 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:32.675189 5345 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47258 (request call id 1798) took 1100 ms. Trace:
I20260929 14:12:32.676869 5342 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:32.677668 5342 raft_consensus.cc:595] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:32.677392 5164 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:32.677237 5345 rpcz_store.cc:276] 0929 14:12:31.574649 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:31.664966 (+ 90317us) service_pool.cc:227] Handling call
0929 14:12:31.665435 (+ 469us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:31.666796 (+ 1361us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:31.666836 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:32.143515 (+476679us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b2c000339d8. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda3450e11 00007fbda3460442 00007fbda4aa8c11 00007fbda4a6d58a 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772
0929 14:12:32.143887 (+ 372us) log.cc:842] Serialized 26 byte log entry
0929 14:12:32.383863 (+239976us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:12:32.383913 (+ 50us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:32.385648 (+ 1735us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:32.674815 (+289167us) raft_consensus.cc:1671] finished
0929 14:12:32.674847 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:32.675112 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":9398,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":1503,"spinlock_wait_cycles":26239744,"thread_start_us":239683,"threads_started":1,"wal-append.queue_time_us":528798}
I20260929 14:12:32.631372 7321 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:32.569844 8045 consensus_queue.cc:237] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:32.730754 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 71082.1us
I20260929 14:12:32.648458 7765 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:32.676525 8230 log.cc:924] Time spent T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.062s user 0.001s sys 0.000s
I20260929 14:12:32.733363 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:32.735760 5316 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:32.736327 8213 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election won for term 6
W20260929 14:12:32.751019 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 233145us
I20260929 14:12:32.622634 8143 consensus_queue.cc:237] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.723246 8236 consensus_queue.cc:1035] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260929 14:12:32.750123 8191 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:32.737973 8072 consensus_queue.cc:1035] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260929 14:12:32.762279 5314 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:32.760665 8213 raft_consensus.cc:686] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:32.772190 8149 raft_consensus.cc:2726] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260929 14:12:32.755491 8068 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:32.773192 8213 consensus_queue.cc:237] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:32.784144 8100 consensus_meta.cc:287] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.532s user 0.004s sys 0.009s
I20260929 14:12:32.784736 8100 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.786607 8154 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:32.744760 7765 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.787365 8100 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:32.788286 5201 raft_consensus.cc:1226] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:32.788792 7765 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:32.789654 8051 raft_consensus.cc:3032] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Stepping down as leader of term 8
I20260929 14:12:32.790167 8051 raft_consensus.cc:729] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:32.791553 8051 consensus_queue.cc:260] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.794324 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:32.766278 8072 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:32.852988 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 286134us
W20260929 14:12:32.853389 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 232602us
I20260929 14:12:32.855983 8154 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:32.857683 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:32.856514 8154 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:32.853232 8100 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:32.860958 8100 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:32.861707 8154 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:32.862977 8100 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:32.864321 7765 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:32.864782 7765 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:32.866573 7765 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:32.859869 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:32.867298 7765 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:32.867792 7765 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:32.869855 7765 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:32.873869 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:32.878963 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 295919us
W20260929 14:12:32.879105 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 109930us
I20260929 14:12:32.879684 5165 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:32.883984 5345 raft_consensus.cc:1226] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260929 14:12:32.884056 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:32.886240 5165 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:32.886792 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:32.888865 5165 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:32.891182 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:32.891773 5275 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:32.894035 5314 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:32.897115 5242 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:32.898130 5242 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:32.903225 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:32.911155 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:32.906498 5242 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:32.913846 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:32.809707 8068 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.888971 8128 raft_consensus.cc:2726] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260929 14:12:32.889863 7765 raft_consensus.cc:2726] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:32.941648 5275 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:32.895761 7976 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:32.858474 8072 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:32.895785 8246 raft_consensus.cc:2781] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:32.896710 8130 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:32.906026 8242 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:32.912024 7996 raft_consensus.cc:2726] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:32.920223 8212 raft_consensus.cc:2726] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:32.918314 8240 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:32.963104 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 68059us
I20260929 14:12:32.956745 8130 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.972494 8068 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:32.952790 8246 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:32.973295 8038 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:32.958359 8258 raft_consensus.cc:2781] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260929 14:12:32.992991 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 96360.9us
W20260929 14:12:32.995709 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 89323.8us
W20260929 14:12:33.007644 8218 log.cc:924] Time spent T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.438s user 0.003s sys 0.000s
I20260929 14:12:33.010064 8038 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:33.010330 8140 raft_consensus.cc:979] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:33.010852 8242 raft_consensus.cc:1067] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:33.013537 5873 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:33.014320 8130 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:33.014942 8254 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:33.014271 5242 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:33.016736 8244 raft_consensus.cc:2726] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:32.986243 8240 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:32.989965 8246 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:33.017204 5242 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:33.027702 8190 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.028970 8189 raft_consensus.cc:2726] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:33.030211 5242 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:33.031508 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:33.018994 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:33.032172 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:33.032472 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:33.021792 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 182911us
I20260929 14:12:33.033090 5349 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:33.032608 5351 raft_consensus.cc:1261] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260929 14:12:33.016665 7321 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.035117 5239 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:33.086690 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:33.086690 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:33.087379 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 57238.9us
I20260929 14:12:33.041162 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:33.088960 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 53895.2us
I20260929 14:12:33.037868 5193 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:12:33.093297 5193 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:33.093799 5193 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:33.095026 7321 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:33.095505 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 45706.9us
I20260929 14:12:33.088176 5242 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:33.098117 5242 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:33.099077 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:33.099725 5270 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 9.
I20260929 14:12:33.100785 5275 raft_consensus.cc:1261] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:33.039069 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:33.040717 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:33.023216 8240 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:33.090606 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:33.091723 5349 raft_consensus.cc:1261] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:33.103925 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:33.118920 5275 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 9
I20260929 14:12:33.113245 5193 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.022270 8072 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:33.088651 8130 raft_consensus.cc:2726] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:33.021287 8258 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:33.124020 5164 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:33.124099 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:33.138455 5344 raft_consensus.cc:3032] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:33.088769 8212 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:33.138873 5344 raft_consensus.cc:729] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:33.129282 5275 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 9.
I20260929 14:12:33.122334 5193 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:33.102257 8261 raft_consensus.cc:2781] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:33.102852 8234 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:33.147291 5344 consensus_queue.cc:260] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.102392 8259 raft_consensus.cc:2726] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:33.177974 5344 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:33.150104 8231 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260929 14:12:33.017638 8244 consensus_queue.cc:1035] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.127s
W20260929 14:12:33.175511 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 244482us
I20260929 14:12:33.114231 8038 consensus_queue.cc:1035] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260929 14:12:33.116750 8233 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.144203 8258 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:33.169731 8261 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:33.150101 8264 raft_consensus.cc:2741] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260929 14:12:33.145937 8100 consensus_queue.cc:1035] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.064s
I20260929 14:12:33.158041 8212 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.215359 8261 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:33.176697 8234 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.188777 8255 consensus_queue.cc:1046] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260929 14:12:33.308311 8246 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.311312 8246 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:33.262879 8262 log.cc:924] Time spent T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.055s user 0.001s sys 0.000s
I20260929 14:12:33.311714 8258 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.259722 8233 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:33.237010 8266 log.cc:924] Time spent T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.087s user 0.000s sys 0.001s
I20260929 14:12:33.314031 8258 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:33.326445 8261 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.328140 8261 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:33.383621 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 268823us
I20260929 14:12:33.411319 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:33.422019 5196 raft_consensus.cc:1226] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 10. Current term is 11. Ops: []
I20260929 14:12:33.231612 8190 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:33.433523 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 335286us
I20260929 14:12:33.435007 8212 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:33.296280 8234 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:33.437664 8115 consensus_queue.cc:1159] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" responder_term: 9 status { last_received { term: 7 index: 4 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9. Ops: [8.5-8.5]" } } last_received_current_leader { term: 0 index: 0 } }
I20260929 14:12:33.447445 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:33.452055 5193 raft_consensus.cc:1226] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:33.422058 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:33.471597 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:33.603571 5349 raft_consensus.cc:2379] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 9: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:33.604276 5314 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:33.495555 5342 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:33.471025 8270 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:33.607363 8191 consensus_queue.cc:1035] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:12:33.608186 8270 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.599823 8261 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:33.610594 8261 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.611245 8270 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:33.611920 5314 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:33.602183 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:33.613461 8222 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election won for term 9
I20260929 14:12:33.614207 8222 raft_consensus.cc:686] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
W20260929 14:12:33.565093 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 129535us
W20260929 14:12:33.564630 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 115795us
I20260929 14:12:33.540844 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:33.555681 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:33.514510 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:33.626449 5270 raft_consensus.cc:1226] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:33.601715 8278 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:33.599607 8135 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:33.631094 8135 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.630903 8278 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:33.634481 8222 consensus_queue.cc:237] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.641489 8135 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:33.649367 8272 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260929 14:12:33.649192 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:33.665675 8242 consensus_queue.cc:1046] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260929 14:12:33.594194 5316 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:33.667506 8261 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:33.667819 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:33.668808 5271 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 9.
I20260929 14:12:33.671983 5348 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:33.662894 8232 raft_consensus.cc:3032] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:33.716125 5314 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:33.719825 8245 raft_consensus.cc:2726] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260929 14:12:33.720036 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 180666us
I20260929 14:12:33.554082 8128 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:33.701510 8232 raft_consensus.cc:729] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
W20260929 14:12:33.741875 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 55669.2us
I20260929 14:12:33.749455 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:33.740401 8135 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260929 14:12:33.751065 8135 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:33.751461 8135 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:33.759281 5196 raft_consensus.cc:1226] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
W20260929 14:12:33.759434 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 127922us
I20260929 14:12:33.763078 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:33.763934 8232 consensus_queue.cc:260] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.766048 8051 raft_consensus.cc:3037] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:33.767542 5200 raft_consensus.cc:2379] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 9: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:33.770340 5242 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:33.770831 8128 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:33.773465 8261 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:33.775890 8222 consensus_queue.cc:1046] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260929 14:12:33.790781 5351 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.557s user 0.017s sys 0.037s
I20260929 14:12:33.790694 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:33.791309 5351 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.673s user 0.017s sys 0.038s
I20260929 14:12:33.769299 5275 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 9.
I20260929 14:12:33.809906 5239 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:33.786378 8245 consensus_queue.cc:1046] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260929 14:12:33.811851 8245 raft_consensus.cc:3032] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 LEADER]: Stepping down as leader of term 10
I20260929 14:12:33.812265 8245 raft_consensus.cc:729] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:33.813478 8278 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:33.814030 8261 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:33.814857 8261 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:33.816390 5871 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.504s user 0.008s sys 0.031s
I20260929 14:12:33.812134 8168 raft_consensus.cc:2781] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260929 14:12:33.817366 5344 consensus_meta.cc:287] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.629s user 0.004s sys 0.012s
I20260929 14:12:33.817497 8168 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:33.817899 5344 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:33.818035 8168 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:33.822757 5342 raft_consensus.cc:595] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:33.823179 5342 raft_consensus.cc:603] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:12:33.823525 8231 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:33.824210 8231 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.825579 8261 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:33.827610 8128 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:33.829872 8168 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.832357 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:33.833127 5346 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 7.
I20260929 14:12:33.832995 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:33.836261 8290 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:33.837038 8290 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.836447 8168 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:33.846909 5343 raft_consensus.cc:3032] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:33.847357 5343 raft_consensus.cc:729] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:33.848296 5343 consensus_queue.cc:260] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.850486 5199 raft_consensus.cc:1226] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:33.853013 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:33.855588 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:33.860455 5164 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:33.860746 5316 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:33.862144 8278 consensus_queue.cc:1046] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260929 14:12:33.863765 8286 log.cc:924] Time spent T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.258s user 0.000s sys 0.002s
I20260929 14:12:33.863924 8242 raft_consensus.cc:3032] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:33.864400 8242 raft_consensus.cc:729] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:33.865253 5314 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:33.865342 8242 consensus_queue.cc:260] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.866717 8242 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:33.868284 8277 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260929 14:12:33.868737 8277 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:33.869145 8277 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:33.872735 5273 raft_consensus.cc:1261] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260929 14:12:33.863765 8278 raft_consensus.cc:2726] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260929 14:12:33.878568 8284 log.cc:924] Time spent T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.277s user 0.001s sys 0.001s
I20260929 14:12:33.879134 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:33.882750 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260929 14:12:33.887713 5350 raft_consensus.cc:1261] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:33.864001 8282 raft_consensus.cc:2781] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:33.892464 8282 raft_consensus.cc:686] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:33.893658 8282 consensus_queue.cc:237] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.895001 8232 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:33.904066 8140 consensus_queue.cc:1035] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:12:33.908233 5239 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:33.909933 8189 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260929 14:12:33.910342 8292 log.cc:924] Time spent T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.064s user 0.002s sys 0.000s
I20260929 14:12:33.912963 8038 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:33.913641 8038 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.914992 8274 raft_consensus.cc:3032] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:33.915434 8274 raft_consensus.cc:729] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:33.915766 8038 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:33.917903 8272 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:33.918380 8272 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:33.921841 5274 raft_consensus.cc:1226] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:33.925113 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:33.925868 5349 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:33.929801 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:12:33.930845 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:33.931108 5343 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:33.892544 8231 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:33.933387 8149 consensus_queue.cc:1159] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" responder_term: 6 status { last_received { term: 6 index: 4 } last_committed_idx: 4 last_received_current_leader { term: 6 index: 4 } }
I20260929 14:12:33.937914 8290 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:33.910356 8189 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:33.940023 8189 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:33.944358 8135 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.916478 8274 consensus_queue.cc:260] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:33.945958 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 85295us
W20260929 14:12:33.947088 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 51235.5us
I20260929 14:12:33.954304 8274 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:33.955092 8245 consensus_queue.cc:260] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:33.956550 8245 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Advancing to term 11
W20260929 14:12:33.958945 8243 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.516s user 0.011s sys 0.014s
W20260929 14:12:33.959334 8243 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.741s user 0.011s sys 0.015s
I20260929 14:12:33.985399 8168 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:33.985831 8264 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:33.986219 8168 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.986706 8264 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:33.990747 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:33.993752 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:33.998070 5272 raft_consensus.cc:1261] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260929 14:12:33.999083 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:33.997054 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:34.001464 5348 raft_consensus.cc:1226] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 9. Current term is 10. Ops: []
I20260929 14:12:34.002684 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:34.003245 5273 raft_consensus.cc:3037] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
W20260929 14:12:34.005692 8295 log.cc:924] Time spent T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.159s user 0.003s sys 0.000s
I20260929 14:12:33.988879 8168 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:34.021443 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 48990.8us
I20260929 14:12:34.027218 5343 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:34.021231 8264 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:34.032933 5271 raft_consensus.cc:1261] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:34.028381 5349 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 7.
I20260929 14:12:34.041680 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:33.960781 8270 consensus_queue.cc:1159] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "675b77a0a88a4bf08134281fc0cd949e" responder_term: 10 status { last_received { term: 10 index: 9 } last_committed_idx: 9 last_received_current_leader { term: 10 index: 9 } }
I20260929 14:12:34.044967 8290 consensus_queue.cc:1035] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20260929 14:12:34.054600 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:34.055771 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:34.056042 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:34.056392 5348 raft_consensus.cc:1261] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260929 14:12:34.070714 5200 raft_consensus.cc:2379] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 6: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:34.056751 8234 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:34.057322 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:34.081516 5273 raft_consensus.cc:2454] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:34.093302 5269 raft_consensus.cc:2379] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 6: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:34.080181 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:34.074199 5239 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:34.098558 5343 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47252 (request call id 2190) took 1426 ms. Trace:
I20260929 14:12:34.104838 5271 raft_consensus.cc:1226] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:34.106642 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:34.116568 5316 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:34.118921 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:34.114362 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:34.098968 5343 rpcz_store.cc:276] 0929 14:12:32.649307 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:32.651530 (+ 2223us) service_pool.cc:227] Handling call
0929 14:12:33.846598 (+1195068us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b54001313a0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda49fd730 00007fbda4a70eff 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:34.075739 (+229141us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":435365632}
I20260929 14:12:34.119860 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:34.121797 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:34.124360 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:34.130702 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:34.130913 5097 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44250 (request call id 127) took 7219 ms. Trace:
I20260929 14:12:34.131974 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:34.133550 5273 raft_consensus.cc:1261] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260929 14:12:34.133831 5273 pending_rounds.cc:76] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5: Aborting all ops after (but not including) 4
I20260929 14:12:34.134058 5273 pending_rounds.cc:97] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260929 14:12:34.134253 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:34.134886 5271 raft_consensus.cc:3037] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:34.134325 5273 raft_consensus.cc:2866] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:12:34.136780 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:34.124354 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:34.142674 8194 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:34.143230 8194 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:34.143741 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:34.104895 8189 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:34.145358 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 56283.1us
I20260929 14:12:34.146854 8189 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:34.138667 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:34.141337 8237 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260929 14:12:34.141685 8148 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:34.154079 8148 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:34.155536 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:34.158660 5196 raft_consensus.cc:1261] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260929 14:12:34.159567 8255 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:34.160387 8265 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:34.160219 8248 consensus_queue.cc:1035] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:34.161000 8265 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:34.103206 8277 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:34.163131 8265 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:34.163738 5345 raft_consensus.cc:1226] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 9. Current term is 10. Ops: []
I20260929 14:12:34.164299 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:34.165879 5272 raft_consensus.cc:1261] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:34.169035 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:34.144944 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:34.177656 8135 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:34.178568 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 48262.2us
I20260929 14:12:34.179102 8277 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:34.180516 8231 consensus_queue.cc:1035] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:12:34.181036 8148 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:34.181747 8290 consensus_queue.cc:1046] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 14:12:34.182755 8290 raft_consensus.cc:3032] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Stepping down as leader of term 9
I20260929 14:12:34.183077 8290 raft_consensus.cc:729] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:34.183980 8290 consensus_queue.cc:260] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:34.142081 8214 raft_consensus.cc:2781] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:34.142261 8125 consensus_queue.cc:1046] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260929 14:12:34.183035 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:34.196832 5271 raft_consensus.cc:2454] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:34.205247 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:34.207010 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:34.207569 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:34.185117 8290 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:34.211319 5164 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:34.212113 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:34.213588 5351 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 2197) took 1177 ms. Trace:
I20260929 14:12:34.218695 5271 raft_consensus.cc:1226] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:34.214538 5275 raft_consensus.cc:2379] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 9: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:34.111244 8234 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:34.206897 8214 raft_consensus.cc:686] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:34.131335 5097 rpcz_store.cc:276] 0929 14:12:26.907769 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:26.907927 (+ 158us) service_pool.cc:227] Handling call
0929 14:12:34.126798 (+7218871us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:12:32.587550 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:12:32.740633 (+153083us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:12:32.743252 (+ 2619us) write_op.cc:435] Acquired schema lock
0929 14:12:32.747638 (+ 4386us) tablet.cc:660] Decoding operations
0929 14:12:32.793573 (+ 45935us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:12:32.793643 (+ 70us) write_op.cc:641] Partition lock acquired for write op
0929 14:12:32.793667 (+ 24us) tablet.cc:683] Acquiring locks for 4 operations
0929 14:12:32.794047 (+ 380us) tablet.cc:699] Row locks acquired
0929 14:12:32.794066 (+ 19us) write_op.cc:260] PREPARE: finished
0929 14:12:32.794204 (+ 138us) write_op.cc:270] Start()
0929 14:12:32.794310 (+ 106us) write_op.cc:276] Timestamp: P: 1790691152794161 usec, L: 0
0929 14:12:32.794329 (+ 19us) op_driver.cc:331] REPLICATION: starting
0929 14:12:32.862365 (+ 68036us) log.cc:842] Serialized 1674 byte log entry
0929 14:12:33.020839 (+158474us) op_driver.cc:444] REPLICATION: finished
0929 14:12:33.320085 (+299246us) write_op.cc:301] APPLY: starting
0929 14:12:33.320173 (+ 88us) tablet.cc:1365] starting BulkCheckPresence
0929 14:12:33.320314 (+ 141us) tablet.cc:1368] finished BulkCheckPresence
0929 14:12:33.320492 (+ 178us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:12:33.321595 (+ 1103us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:12:33.321885 (+ 290us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0929 14:12:33.321958 (+ 73us) write_op.cc:312] APPLY: finished
0929 14:12:33.322270 (+ 312us) log.cc:842] Serialized 41 byte log entry
0929 14:12:33.322475 (+ 205us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:33.322613 (+ 138us) write_op.cc:454] Released schema lock
0929 14:12:33.322672 (+ 59us) write_op.cc:341] FINISH: Updating metrics
0929 14:12:33.323047 (+ 375us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:33.323094 (+ 47us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":551557,"reported_tablets":22,"child_traces":[["op",{"apply.queue_time_us":295173,"apply.run_cpu_time_us":2948,"apply.run_wall_time_us":6204,"num_ops":4,"prepare.queue_time_us":317557,"prepare.run_cpu_time_us":34203,"prepare.run_wall_time_us":358309,"raft.queue_time_us":103245,"raft.run_cpu_time_us":44160,"raft.run_wall_time_us":194056,"replication_time_us":226412,"spinlock_wait_cycles":15104,"thread_start_us":534702,"threads_started":4,"wal-append.queue_time_us":53039}]]}
I20260929 14:12:34.259130 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:34.217166 5351 rpcz_store.cc:276] 0929 14:12:33.036161 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:33.036258 (+ 97us) service_pool.cc:227] Handling call
0929 14:12:33.036532 (+ 274us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:33.791870 (+755338us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:33.791913 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:33.792426 (+ 513us) log.cc:842] Serialized 26 byte log entry
0929 14:12:34.071773 (+279347us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:34.071825 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:34.103453 (+ 31628us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:34.213374 (+109921us) raft_consensus.cc:1671] finished
0929 14:12:34.213391 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:34.213525 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98503,"raft.queue_time_us":93832,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":10692,"thread_start_us":951934,"threads_started":2,"wal-append.queue_time_us":418955}
I20260929 14:12:34.442062 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:34.460779 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:34.457949 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 259625us
W20260929 14:12:34.373834 8295 log.cc:924] Time spent T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.097s user 0.001s sys 0.000s
I20260929 14:12:34.473820 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:34.460851 8189 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:34.446079 8051 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:34.461545 8272 consensus_meta.cc:287] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.543s user 0.008s sys 0.003s
I20260929 14:12:34.258522 8194 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:34.476438 5192 raft_consensus.cc:1226] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9. Ops: []
W20260929 14:12:34.482167 8282 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s user 0.016s sys 0.025s
I20260929 14:12:34.398267 5098 catalog_manager.cc:5495] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:34.495522 8214 consensus_queue.cc:237] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:34.521569 8282 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.544s user 0.017s sys 0.025s
I20260929 14:12:34.530467 8213 raft_consensus.cc:2781] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election won for term 7
W20260929 14:12:34.531050 8262 log.cc:924] Time spent T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20260929 14:12:34.531546 7765 raft_consensus.cc:3032] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Stepping down as leader of term 9
I20260929 14:12:34.531965 7765 raft_consensus.cc:729] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:34.533016 7765 consensus_queue.cc:260] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:34.534178 8125 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:34.534880 8125 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:34.534276 7765 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:34.490500 8189 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:34.552235 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 405247us
I20260929 14:12:34.537159 8125 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:34.562079 5349 raft_consensus.cc:1226] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:34.503288 8272 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:34.582288 8272 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:34.571630 5870 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:34.564143 8189 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:34.584167 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:12:34.553553 8233 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s user 0.012s sys 0.010s
I20260929 14:12:34.585642 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:34.594893 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 101225us
I20260929 14:12:34.547394 8213 raft_consensus.cc:686] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:34.522459 8051 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:34.600841 8254 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:34.605405 5267 raft_consensus.cc:1261] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260929 14:12:34.702200 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:34.737594 8234 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:34.760753 8305 log.cc:924] Time spent T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.079s user 0.000s sys 0.001s
I20260929 14:12:34.752030 8236 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:34.762732 8236 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:34.749393 8233 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:34.779556 8233 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:34.781522 8233 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:34.748124 8213 consensus_queue.cc:237] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:34.794234 8236 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:34.813942 5242 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:34.816493 5201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 2067) took 1014 ms. Trace:
I20260929 14:12:34.823680 8261 raft_consensus.cc:2726] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:34.831679 5349 raft_consensus.cc:1226] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260929 14:12:34.832166 5344 raft_consensus.cc:1226] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 9. Current term is 10. Ops: []
I20260929 14:12:34.832922 5242 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:34.831673 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:34.833236 8319 consensus_queue.cc:1046] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.009s
I20260929 14:12:34.834408 8319 raft_consensus.cc:3032] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:34.834793 8319 raft_consensus.cc:729] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:34.837177 8068 raft_consensus.cc:2726] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:34.838217 5242 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:34.818002 5201 rpcz_store.cc:276] 0929 14:12:33.723133 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:33.752619 (+ 29486us) service_pool.cc:227] Handling call
0929 14:12:33.765682 (+ 13063us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:33.766055 (+ 373us) raft_consensus.cc:1520] Early marking committed up to index 5
0929 14:12:33.985310 (+219255us) log.cc:842] Serialized 14 byte log entry
0929 14:12:34.737144 (+751834us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:34.737237 (+ 93us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:34.737278 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:34.737345 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:34.737464 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":54448,"prepare.queue_time_us":64084,"prepare.run_cpu_time_us":23,"prepare.run_wall_time_us":24,"raft.queue_time_us":66780,"raft.run_cpu_time_us":107,"raft.run_wall_time_us":108,"thread_start_us":871183,"threads_started":3,"wal-append.queue_time_us":594079}
W20260929 14:12:34.851094 8290 consensus_meta.cc:287] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.620s user 0.009s sys 0.002s
W20260929 14:12:34.838254 5165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.677s (SENT) [suppressed 18 similar messages]
I20260929 14:12:34.865288 8259 raft_consensus.cc:2726] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:34.866117 5351 raft_consensus.cc:1261] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260929 14:12:34.867875 8051 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:34.872699 8307 log.cc:924] Time spent T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.144s user 0.001s sys 0.000s
I20260929 14:12:34.877080 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:34.835619 8319 consensus_queue.cc:260] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:34.924628 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:34.926262 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 89498.4us
I20260929 14:12:34.927318 5242 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:34.940289 8259 raft_consensus.cc:2726] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260929 14:12:34.944799 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:34.945595 5276 raft_consensus.cc:2379] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 7: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:34.936401 8319 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
W20260929 14:12:34.957150 8315 log.cc:924] Time spent T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.139s user 0.001s sys 0.000s
I20260929 14:12:34.960280 5242 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:34.971809 5200 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 10.
W20260929 14:12:34.992092 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 235314us
W20260929 14:12:34.973891 8317 log.cc:924] Time spent T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260929 14:12:34.997049 5242 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:34.998016 8189 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:12:35.017767 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:35.028609 5242 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:35.029731 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:35.030398 5274 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:35.010869 8261 raft_consensus.cc:2726] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260929 14:12:35.033856 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:35.034503 5269 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:35.035916 5314 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:35.036149 8236 consensus_queue.cc:1046] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:12:35.037400 8234 raft_consensus.cc:2726] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260929 14:12:35.038341 5314 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:35.039058 8154 raft_consensus.cc:3032] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Stepping down as leader of term 9
I20260929 14:12:35.038975 8234 consensus_queue.cc:1035] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:35.039515 8154 raft_consensus.cc:729] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:35.040385 8154 consensus_queue.cc:260] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.040787 7976 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:35.045482 8236 consensus_queue.cc:1035] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:12:35.047067 8154 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:35.051406 5314 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:35.053496 8234 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:35.053865 8234 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:35.054193 8234 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:35.056473 5316 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:35.058116 8272 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader election won for term 10
I20260929 14:12:35.058655 8272 raft_consensus.cc:686] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:35.059549 8272 consensus_queue.cc:237] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:35.072264 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 51355.6us
I20260929 14:12:35.074357 5274 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 10.
I20260929 14:12:35.075726 5314 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:35.083190 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:35.071900 8261 raft_consensus.cc:2726] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:35.097362 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:35.083968 8277 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader election won for term 10
W20260929 14:12:35.119863 5226 heartbeater.cc:644] Failed to heartbeat to 127.3.242.126:41327 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.3.242.126:41327 timed out after 15.000s (SENT)
I20260929 14:12:35.107368 8319 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:35.122745 8319 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:35.125154 8319 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:35.126173 7976 raft_consensus.cc:2726] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:35.127524 8277 raft_consensus.cc:686] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:35.127655 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:35.128345 5350 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 11.
I20260929 14:12:35.128717 8277 consensus_queue.cc:237] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.129655 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:35.130921 5164 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:35.143419 8246 raft_consensus.cc:465] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Not starting pre-election -- already a leader
I20260929 14:12:35.150477 5274 raft_consensus.cc:1261] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260929 14:12:35.162175 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:35.163231 5274 raft_consensus.cc:2379] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 6: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:35.135479 8154 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260929 14:12:35.165933 5267 raft_consensus.cc:1226] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 LEADER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:35.166275 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:35.169493 5873 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:35.152427 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:35.167295 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:35.174695 8221 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:35.177624 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:35.192466 8234 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.193661 5343 raft_consensus.cc:1226] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:35.199025 8319 consensus_queue.cc:1046] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:12:35.200075 8246 raft_consensus.cc:3032] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:35.200428 8246 raft_consensus.cc:729] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:35.201586 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:35.203172 8234 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:35.201256 8246 consensus_queue.cc:260] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.160037 8242 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:35.203996 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:35.209959 5268 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:35.169181 8154 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:35.211652 8242 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.217303 8242 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:35.218650 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:35.221863 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:35.222559 5276 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:35.223702 5314 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:35.227671 8246 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:35.229205 5268 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:35.233011 8337 raft_consensus.cc:2781] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:35.237870 5314 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:35.237383 8154 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Advancing to term 12
I20260929 14:12:35.241926 8234 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:35.242478 8234 raft_consensus.cc:686] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:35.237735 8337 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:35.243641 8234 consensus_queue.cc:237] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.245846 8337 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:35.250658 8321 consensus_queue.cc:1035] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:12:35.253934 8321 consensus_queue.cc:1046] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:12:35.255015 8319 raft_consensus.cc:3032] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:35.255339 8319 raft_consensus.cc:729] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:35.256085 8319 consensus_queue.cc:260] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.263002 8073 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:35.263698 8073 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:35.266758 8073 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:35.266975 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:35.267599 5350 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 11.
I20260929 14:12:35.268769 5870 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:35.265116 8319 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:35.269645 8073 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260929 14:12:35.270167 8073 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:35.263722 5165 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:35.273511 8320 raft_consensus.cc:2726] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:35.277823 8213 consensus_queue.cc:1035] T 58eff9a0aa514df1999bdd22fb513354 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:35.281590 5165 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:35.282438 8236 raft_consensus.cc:2726] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:35.284670 5165 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:35.287462 8319 raft_consensus.cc:2726] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:35.291918 8337 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.292855 8154 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:35.270581 8073 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Advancing to term 11
I20260929 14:12:35.314623 8337 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:35.329463 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 44874.5us
I20260929 14:12:35.330277 5165 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:35.336066 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:35.336704 5269 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:35.338016 5165 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:35.341379 5165 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:35.342687 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:35.347503 8320 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:35.336369 8246 raft_consensus.cc:2726] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:35.355564 8321 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:12:35.363369 5165 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:35.368314 8236 raft_consensus.cc:2726] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:35.369031 8236 consensus_queue.cc:1046] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:12:35.370186 8236 raft_consensus.cc:3032] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Stepping down as leader of term 7
I20260929 14:12:35.370520 8236 raft_consensus.cc:729] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:35.371292 8236 consensus_queue.cc:260] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.372346 8236 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:35.375241 5165 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:35.377787 8246 raft_consensus.cc:2726] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260929 14:12:35.382100 8214 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:35.382643 8214 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.384929 8214 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:35.393925 5165 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:35.398113 5269 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:35.398469 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:35.399070 5344 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:35.399892 5194 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:35.400777 5343 raft_consensus.cc:1261] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260929 14:12:35.400887 5314 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:35.401757 8234 raft_consensus.cc:2781] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:35.402489 8246 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:35.402781 8234 raft_consensus.cc:686] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:35.404201 8234 consensus_queue.cc:237] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.426893 8154 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 12 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:35.430294 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:35.432649 8154 consensus_queue.cc:1035] T e61c0e0d757e43f19868f329ee0b8bec P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:35.433907 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:35.434473 5343 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Advancing to term 12
I20260929 14:12:35.431975 5165 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:35.435865 5097 catalog_manager.cc:5495] T 632bdccb08444a4d89ecc1cfdbab98b2 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:35.439615 5165 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:35.439198 5097 catalog_manager.cc:5495] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 8 to 9, leader changed from 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 9 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:35.440865 5097 catalog_manager.cc:5495] T 0e4a096fb48c46dd95a1d74d34b6e670 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 5 to 6, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 6 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:35.442569 5097 catalog_manager.cc:5495] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 2 to 4, leader changed from 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 4 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:35.445640 5239 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:35.446175 8236 raft_consensus.cc:2726] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260929 14:12:35.446686 8214 raft_consensus.cc:2781] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:35.447151 8214 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:35.447640 8214 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:35.458241 5269 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:35.457506 5343 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 12.
I20260929 14:12:35.432767 8073 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:35.462363 5164 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:35.466558 5903 raft_consensus.cc:3037] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 11 FOLLOWER]: Advancing to term 12
I20260929 14:12:35.467296 8246 raft_consensus.cc:2781] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 12 FOLLOWER]: Leader election won for term 12
I20260929 14:12:35.454008 8154 raft_consensus.cc:2726] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:35.470193 5165 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:35.471097 5165 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:35.475361 8320 raft_consensus.cc:2726] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:35.457293 8264 consensus_queue.cc:1035] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.015s
I20260929 14:12:35.478621 5194 raft_consensus.cc:1261] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260929 14:12:35.473790 8319 raft_consensus.cc:2726] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:35.477108 5343 raft_consensus.cc:1261] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:35.473284 8246 raft_consensus.cc:686] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [term 12 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:35.483944 5165 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:35.489925 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:35.487195 8242 consensus_queue.cc:1035] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260929 14:12:35.491026 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:35.493799 8320 raft_consensus.cc:2726] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:35.494668 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:35.495476 5201 raft_consensus.cc:2396] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
W20260929 14:12:35.486407 5098 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52726 (request call id 227) took 15709 ms (client timeout 15000 ms). Trace:
I20260929 14:12:35.498411 5903 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 12.
I20260929 14:12:35.487035 8319 consensus_queue.cc:1035] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.013s
I20260929 14:12:35.499635 8246 consensus_queue.cc:237] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:35.511478 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:35.507089 5269 raft_consensus.cc:1261] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260929 14:12:35.512421 5201 raft_consensus.cc:2396] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260929 14:12:35.514683 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:35.515501 5194 raft_consensus.cc:2396] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260929 14:12:35.516165 8242 consensus_queue.cc:1035] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 14:12:35.527534 5099 catalog_manager.cc:5495] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 2 to 8. New cstate: current_term: 8 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:35.513023 8214 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.538817 8073 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 11 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:35.539490 8214 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:35.543995 8272 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:35.531266 5099 catalog_manager.cc:5495] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 4 to 6, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 6 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:35.544556 8272 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.547900 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:35.557314 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:35.558671 5342 raft_consensus.cc:2354] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e for earlier term 11. Current term is 12.
I20260929 14:12:35.560712 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:35.566162 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:35.573627 5870 leader_election.cc:398] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 11 election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e for earlier term 11. Current term is 12.
I20260929 14:12:35.583680 5870 leader_election.cc:401] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
W20260929 14:12:35.497716 5098 rpcz_store.cc:269] 0929 14:12:19.773577 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:19.797934 (+ 24357us) service_pool.cc:227] Handling call
0929 14:12:35.483216 (+15685282us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:12:34.837981 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:12:34.945569 (+107588us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:12:34.945635 (+ 66us) write_op.cc:435] Acquired schema lock
0929 14:12:34.945661 (+ 26us) tablet.cc:660] Decoding operations
0929 14:12:34.946183 (+ 522us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:12:34.951255 (+ 5072us) write_op.cc:641] Partition lock acquired for write op
0929 14:12:34.951289 (+ 34us) tablet.cc:683] Acquiring locks for 1 operations
0929 14:12:34.951596 (+ 307us) tablet.cc:699] Row locks acquired
0929 14:12:34.951630 (+ 34us) write_op.cc:260] PREPARE: finished
0929 14:12:34.951811 (+ 181us) write_op.cc:270] Start()
0929 14:12:34.951953 (+ 142us) write_op.cc:276] Timestamp: P: 1790691154951753 usec, L: 0
0929 14:12:34.951995 (+ 42us) op_driver.cc:331] REPLICATION: starting
0929 14:12:34.952650 (+ 655us) log.cc:842] Serialized 538 byte log entry
0929 14:12:35.241730 (+289080us) op_driver.cc:444] REPLICATION: finished
0929 14:12:35.405063 (+163333us) write_op.cc:301] APPLY: starting
0929 14:12:35.405184 (+ 121us) tablet.cc:1365] starting BulkCheckPresence
0929 14:12:35.405392 (+ 208us) tablet.cc:1368] finished BulkCheckPresence
0929 14:12:35.405551 (+ 159us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:12:35.405896 (+ 345us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:12:35.422234 (+ 16338us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0929 14:12:35.422283 (+ 49us) write_op.cc:312] APPLY: finished
0929 14:12:35.422563 (+ 280us) log.cc:842] Serialized 23 byte log entry
0929 14:12:35.422739 (+ 176us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:35.422816 (+ 77us) write_op.cc:454] Released schema lock
0929 14:12:35.422869 (+ 53us) write_op.cc:341] FINISH: Updating metrics
0929 14:12:35.423190 (+ 321us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:35.423227 (+ 37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":212757,"reported_tablets":42,"child_traces":[["op",{"apply.queue_time_us":163026,"apply.run_cpu_time_us":2435,"apply.run_wall_time_us":18445,"num_ops":1,"prepare.queue_time_us":260603,"prepare.run_cpu_time_us":18311,"prepare.run_wall_time_us":247267,"raft.queue_time_us":106973,"raft.run_cpu_time_us":11919,"raft.run_wall_time_us":94818,"replication_time_us":289652,"thread_start_us":440788,"threads_started":4,"wal-append.queue_time_us":149997}]]}
I20260929 14:12:35.587893 5194 raft_consensus.cc:1261] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260929 14:12:35.588438 5194 pending_rounds.cc:76] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd: Aborting all ops after (but not including) 3
I20260929 14:12:35.589993 5194 pending_rounds.cc:97] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260929 14:12:35.590641 5194 raft_consensus.cc:2866] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 14:12:35.589016 8073 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 675b77a0a88a4bf08134281fc0cd949e for earlier term 11. Current term is 12.
I20260929 14:12:35.604749 5099 catalog_manager.cc:5495] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 6 to 7, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 7 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:35.621320 5099 catalog_manager.cc:5495] T 01b3432723044cbab3d1437e32a2efda P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 5 to 8, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 8 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:12:35.626071 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:35.628746 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:35.626890 5197 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:35.630339 5316 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:35.630367 5198 raft_consensus.cc:2379] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 5: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:35.632169 8242 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:35.633886 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:35.636338 5198 raft_consensus.cc:2379] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 5: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:35.636911 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:35.637630 5268 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:35.633285 8242 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:35.638552 8242 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:35.636232 8131 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:35.641088 5099 catalog_manager.cc:5495] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:35.644387 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:35.634702 8272 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:35.645814 8131 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.645347 5099 catalog_manager.cc:5495] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 4 to 5, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 5 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:35.648016 8131 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:35.650064 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:35.648885 5099 catalog_manager.cc:5495] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 2 to 4, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 4 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:35.664337 5242 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:35.658814 8319 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:35.671490 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:35.644367 8261 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:35.672806 8319 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.683761 8319 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:35.685803 8348 log.cc:924] Time spent T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260929 14:12:35.686654 5193 raft_consensus.cc:1261] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:35.688745 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:35.698472 5268 raft_consensus.cc:1261] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260929 14:12:35.695386 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:35.738758 5165 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:35.742882 5198 raft_consensus.cc:1261] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260929 14:12:35.744881 8277 consensus_queue.cc:1035] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20260929 14:12:35.745831 8346 log.cc:924] Time spent T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20260929 14:12:35.695138 8234 consensus_queue.cc:1035] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260929 14:12:35.746112 8338 consensus_queue.cc:1035] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:12:35.697494 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:35.692061 8264 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260929 14:12:35.781626 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 89987.6us
I20260929 14:12:35.761252 8321 raft_consensus.cc:2726] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260929 14:12:35.785038 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 120801us
I20260929 14:12:35.786266 5242 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:35.785914 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:35.680474 8261 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.803151 5242 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:35.786572 8242 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:35.827205 5351 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:35.828869 5242 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:35.830482 5242 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:35.831636 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 50394.9us
I20260929 14:12:35.797335 8131 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:35.827032 8259 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260929 14:12:35.893890 8264 raft_consensus.cc:2726] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260929 14:12:35.893429 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 52295.4us
I20260929 14:12:35.828639 5239 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:35.920348 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 91809.1us
W20260929 14:12:35.912515 8334 log.cc:924] Time spent T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.088s user 0.000s sys 0.004s
W20260929 14:12:35.912285 8315 log.cc:924] Time spent T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.098s user 0.000s sys 0.004s
W20260929 14:12:35.930559 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 121002us
I20260929 14:12:35.934113 5096 catalog_manager.cc:5495] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:35.922881 8264 raft_consensus.cc:2726] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:35.931742 8051 raft_consensus.cc:2781] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:35.955281 8051 raft_consensus.cc:686] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:35.956420 8051 consensus_queue.cc:237] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:35.964529 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 134118us
I20260929 14:12:36.000725 5242 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:36.018923 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:36.019701 5195 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:35.992015 8264 consensus_queue.cc:1035] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260929 14:12:36.046578 8242 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:36.086843 5195 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:36.096206 5351 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 12 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260929 14:12:36.080606 8366 raft_consensus.cc:2726] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:36.089605 8261 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:36.113579 5316 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:36.108469 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 107819us
W20260929 14:12:36.120399 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 118793us
I20260929 14:12:36.122673 8375 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260929 14:12:36.109025 8350 log.cc:924] Time spent T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.188s user 0.000s sys 0.001s
I20260929 14:12:36.128049 5272 raft_consensus.cc:1261] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:36.134651 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:36.142668 8264 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:36.145869 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:36.156635 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:36.125715 8372 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:36.190680 5275 raft_consensus.cc:2354] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6.
I20260929 14:12:36.190794 5165 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:12:36.218869 8322 log.cc:924] Time spent T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260929 14:12:36.218060 8372 raft_consensus.cc:686] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:36.311860 8372 consensus_queue.cc:237] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:36.150564 8264 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:36.392563 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:36.393532 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 179328us
W20260929 14:12:36.486585 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 138710us
W20260929 14:12:36.488930 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 90503.1us
W20260929 14:12:36.523978 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 357164us
I20260929 14:12:36.529425 8045 consensus_queue.cc:1035] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260929 14:12:36.597158 8391 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:36.554358 8370 raft_consensus.cc:2726] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:36.651929 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 77651.3us
I20260929 14:12:36.597599 8264 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:36.665304 8382 log.cc:924] Time spent T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.141s user 0.000s sys 0.001s
W20260929 14:12:36.665817 8385 log.cc:924] Time spent T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.191s user 0.000s sys 0.000s
W20260929 14:12:36.668164 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 506319us
I20260929 14:12:36.652343 8391 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:36.707928 8386 log.cc:924] Time spent T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
I20260929 14:12:36.714128 5272 raft_consensus.cc:1261] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:36.712508 5239 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:36.717586 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:36.718487 5275 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:36.718779 5350 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:36.719635 5350 raft_consensus.cc:595] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:12:36.727300 8393 log.cc:924] Time spent T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260929 14:12:36.758931 5195 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 6
I20260929 14:12:36.723672 8051 raft_consensus.cc:2726] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260929 14:12:36.762542 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 47033us
I20260929 14:12:36.723769 8223 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:36.764223 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:36.766161 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 96594us
I20260929 14:12:36.768375 5342 raft_consensus.cc:2379] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 10: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:36.752374 8319 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:36.761849 8135 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:36.780825 8319 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:36.781612 8135 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:36.782886 8319 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:36.768411 8371 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:36.783885 8371 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:36.784188 8135 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:36.785867 8371 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:36.763576 8372 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:36.779924 8391 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:36.790096 5096 catalog_manager.cc:5495] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 5 to 7, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 7 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:36.789901 5275 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:36.811317 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:36.812692 5195 raft_consensus.cc:1261] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:36.765619 8321 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:36.823215 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:36.823588 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:36.824383 5200 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:36.836025 5349 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "599240a72b454fca841864f021dbbc64"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:36.836800 5349 raft_consensus.cc:595] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:36.838661 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:36.819146 8386 log.cc:924] Time spent T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260929 14:12:36.814872 8371 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:36.818284 7976 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:36.855712 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 102755us
I20260929 14:12:36.862182 5898 raft_consensus.cc:1261] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 12 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260929 14:12:36.814508 8372 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:36.916440 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 75888.4us
I20260929 14:12:36.790092 8223 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:36.926617 5350 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:36.927206 5350 raft_consensus.cc:595] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:36.864740 8372 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:36.926060 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:36.931836 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 201643us
I20260929 14:12:36.835392 8321 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:36.939503 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 98869us
I20260929 14:12:36.934139 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:36.940974 5267 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
W20260929 14:12:36.926286 8352 log.cc:924] Time spent T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260929 14:12:36.956844 5273 raft_consensus.cc:1261] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:36.935029 8391 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:36.875438 7976 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:36.869270 8371 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:37.006043 8246 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:37.025043 8246 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:37.025671 8402 log.cc:924] Time spent T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.192s user 0.001s sys 0.000s
I20260929 14:12:37.025870 5342 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:37.027024 8246 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:36.977396 8404 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:37.042972 8321 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:36.953459 8372 consensus_queue.cc:1035] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20260929 14:12:37.056372 5316 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:36.959265 8367 consensus_queue.cc:1035] T a38ce5b8e5024c0ab37787aaf2b357c0 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.081s
W20260929 14:12:37.030055 8408 log.cc:924] Time spent T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
I20260929 14:12:37.058465 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:37.020566 8319 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:37.069387 8369 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:37.070139 8369 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:37.071590 5344 raft_consensus.cc:1261] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:12:37.073745 8264 consensus_queue.cc:1035] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:12:37.069533 8051 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:37.074756 8051 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:36.982054 8391 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:37.044298 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 279703us
I20260929 14:12:37.083704 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:37.084406 5269 raft_consensus.cc:1261] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260929 14:12:37.057459 8404 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:37.086985 8399 log.cc:924] Time spent T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260929 14:12:37.092499 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:37.091751 8261 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:37.067894 8338 raft_consensus.cc:2781] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:37.068986 8319 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:37.107204 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:37.116887 5343 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 10.
W20260929 14:12:37.203949 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 85433.8us
W20260929 14:12:37.205511 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 95994.7us
I20260929 14:12:37.081674 8051 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:37.099611 8261 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:37.100680 7976 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:37.116521 8338 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:37.205905 8371 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:37.204742 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 69133.4us
W20260929 14:12:37.310849 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 199105us
W20260929 14:12:37.352501 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 143502us
I20260929 14:12:37.352933 8418 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:37.353731 8418 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:37.358321 5200 raft_consensus.cc:1261] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:12:37.358500 8415 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:37.359063 8415 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:37.362512 8261 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:37.365263 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:37.365533 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:37.366434 5195 raft_consensus.cc:2396] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20260929 14:12:37.367106 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 129686us
I20260929 14:12:37.261234 8338 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:37.235003 8372 consensus_queue.cc:1035] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.137s
I20260929 14:12:37.338706 8401 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:37.345290 8371 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:37.342062 8320 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:37.447988 5270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50770 (request call id 2100) took 1313 ms. Trace:
W20260929 14:12:37.452032 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 393862us
I20260929 14:12:37.455227 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:37.463085 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 120535us
W20260929 14:12:37.447842 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 131628us
I20260929 14:12:37.467279 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:37.508085 5096 catalog_manager.cc:5495] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 6 to 8, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 8 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:37.510329 5096 catalog_manager.cc:5495] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:37.511729 5096 catalog_manager.cc:5495] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:37.450323 5270 rpcz_store.cc:276] 0929 14:12:36.118986 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:36.119103 (+ 117us) service_pool.cc:227] Handling call
0929 14:12:36.147419 (+ 28316us) raft_consensus.cc:1457] Updating replica for 2 ops
0929 14:12:36.159145 (+ 11726us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:36.159176 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0929 14:12:36.392771 (+233595us) log.cc:842] Serialized 52 byte log entry
0929 14:12:36.720459 (+327688us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:12:36.797410 (+ 76951us) log.cc:842] Serialized 14 byte log entry
0929 14:12:37.419508 (+622098us) log.cc:842] Serialized 14 byte log entry
0929 14:12:37.432450 (+ 12942us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:37.432551 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:37.432608 (+ 57us) raft_consensus.cc:1671] finished
0929 14:12:37.432625 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:37.432796 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1246982,"prepare.queue_time_us":33561,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":34,"raft.queue_time_us":91489,"raft.run_cpu_time_us":421,"raft.run_wall_time_us":26645,"thread_start_us":270880,"threads_started":1,"wal-append.queue_time_us":364193}
I20260929 14:12:37.453963 8371 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:37.449810 8401 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:37.458509 8320 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:37.550554 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:37.521102 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:37.587325 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:37.564738 5096 catalog_manager.cc:5495] T e50b67ada64545a6b7525d4aba317bb0 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:37.590176 5200 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:37.564316 5099 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52736 (request call id 225) took 8380 ms. Trace:
I20260929 14:12:37.590771 5096 catalog_manager.cc:5495] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 7 to 9, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 9 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:37.594830 5344 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:37.595629 5344 raft_consensus.cc:595] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:37.499586 8369 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:37.420707 8223 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:37.249035 8404 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:37.361045 8415 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:37.356307 8418 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:37.623430 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:37.624351 5198 raft_consensus.cc:2396] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260929 14:12:37.624866 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:37.620235 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:37.600309 5096 catalog_manager.cc:5495] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:37.627592 5165 leader_election.cc:398] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6.
I20260929 14:12:37.627272 5274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50770 (request call id 2101) took 1508 ms. Trace:
I20260929 14:12:37.628103 5165 leader_election.cc:401] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:12:37.630005 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:37.630982 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 276357us
I20260929 14:12:37.631649 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:37.521152 8375 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:37.547890 8391 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:37.593906 5099 rpcz_store.cc:276] 0929 14:12:29.184310 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:29.186404 (+ 2094us) service_pool.cc:227] Handling call
0929 14:12:37.564219 (+8377815us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:12:35.682627 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:12:35.683292 (+ 665us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:12:35.683340 (+ 48us) write_op.cc:435] Acquired schema lock
0929 14:12:35.683364 (+ 24us) tablet.cc:660] Decoding operations
0929 14:12:35.684455 (+ 1091us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:12:35.684526 (+ 71us) write_op.cc:641] Partition lock acquired for write op
0929 14:12:35.684563 (+ 37us) tablet.cc:683] Acquiring locks for 7 operations
0929 14:12:35.685297 (+ 734us) tablet.cc:699] Row locks acquired
0929 14:12:35.685315 (+ 18us) write_op.cc:260] PREPARE: finished
0929 14:12:35.685561 (+ 246us) write_op.cc:270] Start()
0929 14:12:35.685673 (+ 112us) write_op.cc:276] Timestamp: P: 1790691155685514 usec, L: 0
0929 14:12:35.685692 (+ 19us) op_driver.cc:331] REPLICATION: starting
0929 14:12:35.686265 (+ 573us) log.cc:842] Serialized 2849 byte log entry
0929 14:12:35.698579 (+ 12314us) op_driver.cc:444] REPLICATION: finished
0929 14:12:35.698939 (+ 360us) write_op.cc:301] APPLY: starting
0929 14:12:35.699015 (+ 76us) tablet.cc:1365] starting BulkCheckPresence
0929 14:12:35.699193 (+ 178us) tablet.cc:1368] finished BulkCheckPresence
0929 14:12:35.699280 (+ 87us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:12:35.700062 (+ 782us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:12:35.700236 (+ 174us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0929 14:12:35.700279 (+ 43us) write_op.cc:312] APPLY: finished
0929 14:12:35.700498 (+ 219us) log.cc:842] Serialized 59 byte log entry
0929 14:12:35.700646 (+ 148us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:35.700754 (+ 108us) write_op.cc:454] Released schema lock
0929 14:12:35.700792 (+ 38us) write_op.cc:341] FINISH: Updating metrics
0929 14:12:35.701040 (+ 248us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:35.701078 (+ 38us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":19620,"reported_tablets":34,"spinlock_wait_cycles":1409408,"child_traces":[["op",{"apply.queue_time_us":188,"apply.run_cpu_time_us":2508,"apply.run_wall_time_us":35280,"num_ops":7,"prepare.queue_time_us":652,"prepare.run_cpu_time_us":3954,"prepare.run_wall_time_us":12353,"replication_time_us":12803}]]}
I20260929 14:12:37.560946 8319 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:37.636685 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:37.650069 8375 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:37.635974 8418 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:37.653254 8418 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:37.655185 8418 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:37.655314 8375 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:37.628257 5274 rpcz_store.cc:276] 0929 14:12:36.120078 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:36.120274 (+ 196us) service_pool.cc:227] Handling call
0929 14:12:36.120684 (+ 410us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:36.122250 (+ 1566us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:36.122275 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:36.122714 (+ 439us) log.cc:842] Serialized 26 byte log entry
0929 14:12:37.310555 (+1187841us) spinlock_profiling.cc:243] Waited 1.04 ms on lock 0x7b18000003a0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda46fc82c 00007fbda1b279dc 00007fbdaa95ea39 00007fbda1b448bb 00007fbda1b42cbf 00007fbda1b44e22 00007fbda46ee5bb 00007fbda46f4063 00007fbda46f4411
0929 14:12:37.310736 (+ 181us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:37.611719 (+300983us) log.cc:842] Serialized 14 byte log entry
0929 14:12:37.621550 (+ 9831us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:37.621638 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:37.621697 (+ 59us) raft_consensus.cc:1671] finished
0929 14:12:37.627076 (+ 5379us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:37.627216 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":109693,"raft.queue_time_us":14487,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":9779,"spinlock_wait_cycles":2385792,"thread_start_us":199716,"threads_started":2,"wal-append.queue_time_us":239676}
I20260929 14:12:37.637167 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:37.639240 5096 catalog_manager.cc:5495] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:37.642189 8401 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:37.643843 8392 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:37.629273 8377 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:37.646570 8320 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:37.659658 5314 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:37.661717 5316 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:37.665741 5096 catalog_manager.cc:5495] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 6 to 7, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 7 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:37.630234 8387 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:37.632850 8261 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:37.681082 5096 catalog_manager.cc:5495] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 7 to 9, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 9 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:37.685833 5096 catalog_manager.cc:5495] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 5 to 6, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 6 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:37.687584 5096 catalog_manager.cc:5495] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 7 to 10, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 10 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:37.688386 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:37.689267 5273 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 9.
W20260929 14:12:37.690762 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 349577us
I20260929 14:12:37.720505 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:37.714743 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 53107.7us
I20260929 14:12:37.702076 5096 catalog_manager.cc:5495] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:37.734733 5096 catalog_manager.cc:5495] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:37.721235 5272 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 8.
I20260929 14:12:37.791313 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:37.667387 8051 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:37.683579 8369 raft_consensus.cc:2726] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:37.738735 8372 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:37.722666 8261 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:37.779181 8259 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:37.779774 8338 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:37.672766 8392 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:37.788671 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:37.789395 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 115077us
I20260929 14:12:37.789678 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:37.789980 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:37.790614 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:37.799968 5096 catalog_manager.cc:5495] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 7 to 10, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 10 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:37.798458 5316 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:37.804823 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:37.814237 5164 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:37.817783 5242 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:37.821003 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:37.837531 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:37.838584 5351 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:37.844923 5351 raft_consensus.cc:595] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Received request to transfer leadership to TS ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:37.713721 8387 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:37.675961 8377 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:37.842829 5096 catalog_manager.cc:5495] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 5 to 6, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 6 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:37.855649 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:37.856005 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:37.850150 5164 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:37.858263 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:37.868484 8409 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:37.869035 8409 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:37.869390 8409 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:37.870323 8261 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:37.871006 8261 consensus_queue.cc:1035] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:12:37.872272 8261 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:37.873493 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260929 14:12:37.875098 8364 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:37.875326 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:12:37.876024 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:37.877866 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:37.879729 8369 consensus_queue.cc:1035] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260929 14:12:37.839471 8259 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:37.800652 8370 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:37.811157 8395 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:12:37.889627 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 141803us
W20260929 14:12:38.003137 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 109606us
I20260929 14:12:37.912717 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:37.895787 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:37.944657 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 100121us
I20260929 14:12:37.943931 5275 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 2144) took 1095 ms. Trace:
I20260929 14:12:38.036571 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:38.038188 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:37.919207 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:37.811602 8372 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:38.038329 5275 rpcz_store.cc:276] 0929 14:12:36.848238 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:36.855401 (+ 7163us) service_pool.cc:227] Handling call
0929 14:12:36.857508 (+ 2107us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:36.870507 (+ 12999us) raft_consensus.cc:1520] Early marking committed up to index 6
0929 14:12:37.802176 (+931669us) log.cc:842] Serialized 14 byte log entry
0929 14:12:37.943523 (+141347us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:37.943584 (+ 61us) raft_consensus.cc:1631] Marking committed up to 6
0929 14:12:37.943646 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:37.943727 (+ 81us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:37.943861 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":542746,"prepare.queue_time_us":748022,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":24747,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":97,"thread_start_us":511714,"threads_started":2}
I20260929 14:12:38.045761 5268 raft_consensus.cc:2379] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 in current term 8: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:38.055333 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:38.060488 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:38.052186 5242 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:37.860987 8397 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260929 14:12:38.067924 5242 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:38.059553 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 261143us
I20260929 14:12:37.939157 8420 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6.
I20260929 14:12:38.070065 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:37.986753 8413 raft_consensus.cc:979] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:37.957233 8320 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:37.990455 8259 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:37.986253 8338 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:37.800835 8051 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:38.075254 5239 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:38.174389 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 99446.1us
I20260929 14:12:38.175223 8051 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:38.175845 8131 raft_consensus.cc:2726] T 678eb0db533d4bea872808b6fb39b764 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:37.884274 8272 consensus_queue.cc:1035] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.193s
W20260929 14:12:38.138262 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 70416.3us
I20260929 14:12:38.060729 8372 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:38.165831 8375 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:38.192155 8375 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:38.194499 8375 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:38.070942 8391 raft_consensus.cc:2726] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:37.980801 8392 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:37.907049 8371 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:37.977859 8377 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:38.187521 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:38.199378 5242 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:38.202175 8361 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:38.202751 8361 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:38.072106 8395 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:38.206449 5272 raft_consensus.cc:2454] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:38.209568 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:38.210846 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:38.211970 8395 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:38.205116 8361 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:38.216337 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:38.219023 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:38.219964 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:38.221866 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:38.223871 5201 raft_consensus.cc:1261] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260929 14:12:38.260226 5197 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:38.265794 8409 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:38.267936 8409 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:38.270635 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 57088.8us
I20260929 14:12:38.270852 5197 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:38.225630 5164 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:38.272534 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 46989.9us
I20260929 14:12:38.275238 8375 raft_consensus.cc:2726] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:38.275395 8378 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:38.275959 8378 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:38.228580 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:38.278003 8378 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:38.230604 7976 raft_consensus.cc:979] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:38.054563 8370 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:38.278440 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 56326.3us
I20260929 14:12:38.279891 8370 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:38.076443 8397 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:38.280455 8397 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Advancing to term 11
I20260929 14:12:38.248930 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:38.250478 8366 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:38.286374 8366 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:38.288446 8366 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:38.290071 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:38.291725 8371 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:38.253154 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:38.292222 8371 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:38.293769 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:38.294420 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:38.276177 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 460657us
I20260929 14:12:38.296635 7976 raft_consensus.cc:1067] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Signalling peer c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:38.296260 8397 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:38.232352 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:38.297719 5165 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:38.146773 8379 raft_consensus.cc:2726] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:38.180630 8320 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:38.301954 8320 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:38.303228 8320 raft_consensus.cc:2726] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:38.299414 5165 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:38.310094 8370 raft_consensus.cc:2726] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:38.313155 5239 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:38.316051 8440 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:38.315178 5165 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:38.318239 8387 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:38.318872 8383 raft_consensus.cc:2726] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:37.978451 8431 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:38.320574 8431 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:38.190775 8413 raft_consensus.cc:1067] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Signalling peer c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:38.323904 8397 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:38.325592 8371 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:38.329288 5873 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:38.330778 5873 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:38.332043 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:38.332731 5274 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:38.333418 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:38.332731 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:38.334097 5273 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:38.335522 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:38.338217 8430 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:38.339026 8430 raft_consensus.cc:2726] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:38.340000 8442 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:38.340540 8442 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:38.341526 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 123873us
I20260929 14:12:38.343914 8420 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:38.345544 8420 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:38.346524 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:38.348590 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:38.349880 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:38.328238 8431 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:38.357926 8442 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:38.360268 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:38.361775 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:38.363281 8375 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:38.363871 8375 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:38.363271 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:38.367251 8375 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:38.367940 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:38.368319 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:38.369323 8395 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:38.369851 8395 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:38.371711 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:38.372076 8395 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:38.372422 8379 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:38.368268 8368 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:38.372889 8379 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:38.373230 8368 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:38.374984 8379 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:38.375044 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:38.375073 8368 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:38.379501 5164 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:38.397913 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:38.398213 5164 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:38.386878 5314 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.044s sys 0.080s
I20260929 14:12:38.387280 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:38.399148 5314 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.533s user 0.044s sys 0.080s
I20260929 14:12:38.387897 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:38.390312 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:38.392309 5273 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 9.
W20260929 14:12:38.395964 5096 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.065s sys 0.143s
W20260929 14:12:38.401170 5096 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.524s user 0.065s sys 0.143s
I20260929 14:12:38.382073 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:38.400295 5194 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:38.439806 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:38.441190 5314 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:38.377031 8420 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:38.440689 5194 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 9.
W20260929 14:12:38.493759 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 52682.5us
I20260929 14:12:38.444733 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:38.471468 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:38.522718 5314 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:38.439683 8383 raft_consensus.cc:2726] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:38.535887 8403 raft_consensus.cc:979] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:38.536473 8403 raft_consensus.cc:1067] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Signalling peer c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:38.444146 8370 raft_consensus.cc:2726] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:38.491339 8440 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:38.390743 8261 raft_consensus.cc:2726] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260929 14:12:38.566306 8440 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:38.532637 8338 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:38.614598 5271 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 10.
I20260929 14:12:38.532308 8443 raft_consensus.cc:2726] T 7b630b7184c5456bad1fbd45514e52ba P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:38.719547 8259 raft_consensus.cc:2726] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260929 14:12:38.748029 8390 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.524s user 0.012s sys 0.005s
I20260929 14:12:39.015177 8045 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:38.774900 8370 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:39.017295 8045 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:39.058764 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:39.059669 5272 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:39.060813 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:39.064944 8390 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.792s user 0.013s sys 0.005s
W20260929 14:12:39.065963 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 735193us
I20260929 14:12:39.067358 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:39.067514 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 674744us
I20260929 14:12:39.074277 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:39.074992 5345 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Advancing to term 11
I20260929 14:12:39.082650 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:39.082918 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:39.084388 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:39.020884 8320 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:12:39.091565 5197 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.712s user 0.011s sys 0.004s
W20260929 14:12:39.093345 5197 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.714s user 0.011s sys 0.004s
I20260929 14:12:39.073933 8213 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:39.096834 8213 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:39.098371 8045 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:39.098789 8213 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:39.099332 5268 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:50776
I20260929 14:12:39.099884 5268 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:39.100181 5268 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:39.106132 5271 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50776 (request call id 2152) took 1254 ms. Trace:
I20260929 14:12:39.106806 5268 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:39.108575 5268 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:39.106518 5271 rpcz_store.cc:276] 0929 14:12:37.851983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:37.854637 (+ 2654us) service_pool.cc:227] Handling call
0929 14:12:39.105934 (+1251297us) spinlock_profiling.cc:243] Waited 574 ms on lock 0x7b54002fe9d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:39.106042 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1321083136}
W20260929 14:12:39.114909 8372 consensus_meta.cc:287] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.875s user 0.011s sys 0.000s
I20260929 14:12:39.115365 8372 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:39.117342 5348 raft_consensus.cc:3037] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:39.074661 8370 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:39.123087 8370 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:39.077003 8397 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:39.124806 8440 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:39.124326 8397 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:39.126684 8397 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:39.123087 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 56145.4us
I20260929 14:12:39.095134 5273 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:50776
I20260929 14:12:39.134804 8320 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:39.144482 5273 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:39.145169 5273 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:39.158584 8357 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:39.159325 8357 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:39.168984 5345 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 11.
I20260929 14:12:39.175926 5273 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:39.184118 5272 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:39.185385 5273 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:39.188946 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:39.198169 8372 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:39.199242 8320 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:39.202315 8357 leader_election.cc:288] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:39.219157 5348 raft_consensus.cc:2454] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 9.
I20260929 14:12:39.262970 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:39.266865 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:39.274150 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:39.278708 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:39.309126 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:39.355557 8445 log.cc:924] Time spent T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260929 14:12:39.370684 5276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 2158) took 1517 ms. Trace:
I20260929 14:12:39.371170 5276 rpcz_store.cc:276] 0929 14:12:37.853729 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:37.857306 (+ 3577us) service_pool.cc:227] Handling call
0929 14:12:37.857587 (+ 281us) raft_consensus.cc:1457] Updating replica for 2 ops
0929 14:12:38.056425 (+198838us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:38.056454 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0929 14:12:38.146519 (+ 90065us) log.cc:842] Serialized 52 byte log entry
0929 14:12:38.638458 (+491939us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:38.638813 (+ 355us) log.cc:842] Serialized 14 byte log entry
0929 14:12:38.659599 (+ 20786us) log.cc:842] Serialized 14 byte log entry
0929 14:12:38.680715 (+ 21116us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:38.682931 (+ 2216us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:39.370438 (+687507us) raft_consensus.cc:1671] finished
0929 14:12:39.370464 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:39.370617 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13980,"prepare.queue_time_us":25592,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":79,"raft.queue_time_us":114562,"raft.run_cpu_time_us":369,"raft.run_wall_time_us":6420,"thread_start_us":670634,"threads_started":2,"wal-append.queue_time_us":1095088}
I20260929 14:12:39.379860 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:39.380746 5198 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:39.381695 8261 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:39.385792 8261 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:39.388223 8261 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:39.376547 8370 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:39.376449 8387 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:39.382839 8368 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:39.392764 8415 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:39.392554 8320 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:39.406653 8370 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:39.416255 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:39.422533 5348 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47252 (request call id 2325) took 1133 ms. Trace:
I20260929 14:12:39.423074 5348 rpcz_store.cc:276] 0929 14:12:38.288924 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:38.289808 (+ 884us) service_pool.cc:227] Handling call
0929 14:12:39.422252 (+1132444us) spinlock_profiling.cc:243] Waited 826 ms on lock 0x7b540012d254. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:39.422437 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1900716416}
I20260929 14:12:39.426677 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:39.431303 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:39.432703 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:39.433507 5192 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:39.458657 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:39.460359 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:39.461110 5194 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:39.462119 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:39.462407 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:39.463215 5200 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:39.459609 5201 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 10.
I20260929 14:12:39.465057 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:39.409013 8387 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:39.461823 8370 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:39.414912 8320 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:39.429028 8368 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:39.415094 8415 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:39.502441 5194 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 7.
I20260929 14:12:39.531317 8259 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:12:39.542593 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 470130us
W20260929 14:12:39.540428 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 1.14439e+06us
I20260929 14:12:39.557682 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:39.558701 5348 raft_consensus.cc:3032] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:39.559198 5348 raft_consensus.cc:729] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:39.557440 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:39.569746 5873 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:39.560391 5348 consensus_queue.cc:260] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:39.543768 8440 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:39.583496 5350 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.064s user 0.027s sys 0.008s
W20260929 14:12:39.583925 5350 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.086s user 0.028s sys 0.008s
I20260929 14:12:39.584048 5197 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1790) took 1184 ms. Trace:
I20260929 14:12:39.584475 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:39.574405 5348 raft_consensus.cc:3037] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:39.543790 8337 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:39.574369 5314 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.023s user 0.009s sys 0.022s
W20260929 14:12:39.586927 5314 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.054s user 0.010s sys 0.022s
W20260929 14:12:39.587253 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 1.05436e+06us
I20260929 14:12:39.584766 5350 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 2314) took 1641 ms. Trace:
W20260929 14:12:39.588935 5199 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.906s user 0.006s sys 0.031s
W20260929 14:12:39.589305 5199 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.907s user 0.006s sys 0.031s
I20260929 14:12:39.590205 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52188 (request call id 1786) took 1226 ms. Trace:
I20260929 14:12:39.584440 5197 rpcz_store.cc:276] 0929 14:12:38.376833 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:38.378590 (+ 1757us) service_pool.cc:227] Handling call
0929 14:12:38.378961 (+ 371us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:38.379194 (+ 233us) raft_consensus.cc:1520] Early marking committed up to index 4
0929 14:12:39.093723 (+714529us) log.cc:842] Serialized 14 byte log entry
0929 14:12:39.561017 (+467294us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:39.561079 (+ 62us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:12:39.561135 (+ 56us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:39.561234 (+ 99us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:39.561367 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":826044,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":65,"raft.queue_time_us":15830,"thread_start_us":1166843,"threads_started":2}
I20260929 14:12:39.588016 5350 rpcz_store.cc:276] 0929 14:12:37.943347 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:37.943489 (+ 142us) service_pool.cc:227] Handling call
0929 14:12:38.245338 (+301849us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b5800183998. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:12:38.245530 (+ 192us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:38.245721 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 10
0929 14:12:38.497521 (+251800us) log.cc:842] Serialized 14 byte log entry
0929 14:12:39.584470 (+1086949us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:39.584520 (+ 50us) raft_consensus.cc:1631] Marking committed up to 10
0929 14:12:39.584568 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:39.584629 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:39.584725 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":106569,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":56,"raft.queue_time_us":84,"spinlock_wait_cycles":39222016,"thread_start_us":1338009,"threads_started":2}
I20260929 14:12:39.590647 5199 rpcz_store.cc:276] 0929 14:12:38.364320 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:38.364453 (+ 133us) service_pool.cc:227] Handling call
0929 14:12:38.364761 (+ 308us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:38.365081 (+ 320us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:38.365110 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:38.365542 (+ 432us) log.cc:842] Serialized 26 byte log entry
0929 14:12:38.669556 (+304014us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:12:39.589694 (+920138us) log.cc:842] Serialized 14 byte log entry
0929 14:12:39.589944 (+ 250us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:39.590009 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:39.590044 (+ 35us) raft_consensus.cc:1671] finished
0929 14:12:39.590054 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:39.590157 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":537577,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":93,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":132,"thread_start_us":1210504,"threads_started":2,"wal-append.queue_time_us":867963}
I20260929 14:12:39.599676 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:39.606127 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:39.607057 5350 raft_consensus.cc:3032] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:39.607407 5350 raft_consensus.cc:729] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:39.616542 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:39.575800 7976 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:39.615674 5348 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:39.618160 8261 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:39.574133 8372 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:39.620221 5165 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:39.619016 8415 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:39.618764 7976 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:39.622835 8454 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:39.623431 8456 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:39.639206 5165 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:12:39.653560 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 83920.5us
I20260929 14:12:39.608285 5350 consensus_queue.cc:260] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:39.629781 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae31abec82ff4f40b695ccb19f289dbc" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:39.555497 8259 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:39.628082 5192 raft_consensus.cc:2454] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
W20260929 14:12:39.636952 5316 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.542s user 0.000s sys 0.015s
W20260929 14:12:39.669477 5316 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.577s user 0.000s sys 0.016s
I20260929 14:12:39.670258 8259 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:39.670562 5316 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:39.669804 5350 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:39.681988 8433 raft_consensus.cc:2726] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:39.682235 5165 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:39.692459 5165 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:39.694332 8433 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:39.694928 8433 raft_consensus.cc:2726] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260929 14:12:39.682700 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 44640.1us
W20260929 14:12:39.683625 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RunLeaderElection -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 61727.1us
I20260929 14:12:39.695816 5314 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:39.685470 5316 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:39.685037 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 129580us
I20260929 14:12:39.693809 5165 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:39.705837 8433 raft_consensus.cc:2781] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election won for term 9
I20260929 14:12:39.706377 8433 raft_consensus.cc:686] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:39.707242 8433 consensus_queue.cc:237] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:39.708696 5350 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 7.
I20260929 14:12:39.590394 8440 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:39.719449 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 163997us
I20260929 14:12:39.713529 5164 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:39.620697 8442 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:39.616444 8045 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:39.738211 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:39.738353 8442 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:39.739902 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 55398.7us
I20260929 14:12:39.740346 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:39.728094 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 67348.7us
W20260929 14:12:39.715616 8449 log.cc:924] Time spent T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
I20260929 14:12:39.738452 8045 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:39.594571 8337 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:39.741040 8442 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:39.742689 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:39.743332 5192 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 9.
I20260929 14:12:39.683990 8415 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260929 14:12:39.747265 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 61910.3us
I20260929 14:12:39.750630 5316 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:39.715487 8272 raft_consensus.cc:2726] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260929 14:12:39.762856 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 67115.5us
I20260929 14:12:39.765929 5276 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:50776
I20260929 14:12:39.766474 5276 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:39.766806 5276 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:39.776589 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:39.781553 5272 raft_consensus.cc:2354] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 8. Current term is 9.
I20260929 14:12:39.790035 5273 raft_consensus.cc:1226] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:39.638073 8261 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:39.681074 8372 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:39.747063 8371 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:39.750093 8415 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 14:12:39.803932 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 60432.9us
I20260929 14:12:39.892325 5239 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:39.786136 8338 raft_consensus.cc:2781] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:39.777812 8387 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:39.776997 8045 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:39.786118 8413 raft_consensus.cc:2741] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260929 14:12:39.749601 8135 consensus_queue.cc:1035] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260929 14:12:39.687147 8456 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:39.943941 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 206105us
I20260929 14:12:39.685037 8454 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:39.930748 5239 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:39.802369 8441 raft_consensus.cc:2726] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:39.821740 8320 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:39.941752 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 140382us
I20260929 14:12:39.805858 8359 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:39.966054 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 225526us
I20260929 14:12:39.950582 5242 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:39.969604 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 220376us
I20260929 14:12:39.771763 8474 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:39.821539 8415 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
W20260929 14:12:39.976087 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 45453us
I20260929 14:12:39.979539 5164 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:39.972083 5316 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:12:39.961800 8413 raft_consensus.cc:979] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:39.940433 8045 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:39.939605 8338 raft_consensus.cc:686] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:39.950067 8375 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:40.013300 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 263545us
I20260929 14:12:39.973047 8419 raft_consensus.cc:2781] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Leader election won for term 11
I20260929 14:12:39.938789 8371 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:39.984547 8443 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:39.986407 8368 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:39.991181 8441 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
W20260929 14:12:40.112572 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 45303.1us
I20260929 14:12:40.022431 8413 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:40.124257 8337 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:40.028847 8375 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.124838 8337 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:40.125211 8466 log.cc:924] Time spent T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
I20260929 14:12:40.125384 8337 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.126590 8375 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:40.126776 8479 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:40.127369 8337 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:40.127830 8337 raft_consensus.cc:1067] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Signalling peer ac17d3b553b74c9da39e0ab307addacd to start an election
I20260929 14:12:40.127948 8375 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:40.128602 8337 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:40.105794 8419 raft_consensus.cc:686] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [term 11 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:40.129849 8442 raft_consensus.cc:2781] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:40.130370 8442 raft_consensus.cc:686] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:40.130262 8419 consensus_queue.cc:237] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.131407 8442 consensus_queue.cc:237] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.132329 8469 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:40.132985 8469 raft_consensus.cc:482] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:40.133421 8469 raft_consensus.cc:504] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.134334 8371 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:40.134871 8371 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:40.135201 8371 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:40.135452 8469 leader_election.cc:288] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:40.135553 8371 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:39.994917 8474 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:40.137427 7976 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:40.106159 8455 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:40.140803 8455 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.141585 5270 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 2166) took 1083 ms. Trace:
I20260929 14:12:40.142786 8455 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:40.143359 8456 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:40.144016 8338 consensus_queue.cc:237] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.124778 8368 raft_consensus.cc:2781] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:40.145234 8368 raft_consensus.cc:686] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:40.145915 8454 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:40.128425 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260929 14:12:40.124404 8458 log.cc:924] Time spent T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.140s user 0.000s sys 0.000s
I20260929 14:12:40.141919 5270 rpcz_store.cc:276] 0929 14:12:39.058575 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:39.064296 (+ 5721us) service_pool.cc:227] Handling call
0929 14:12:39.082255 (+ 17959us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:39.533775 (+451520us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:39.533818 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:39.536481 (+ 2663us) log.cc:842] Serialized 26 byte log entry
0929 14:12:39.680959 (+144478us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:39.681038 (+ 79us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:39.681176 (+ 138us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:40.141395 (+460219us) raft_consensus.cc:1671] finished
0929 14:12:40.141415 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:40.141534 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":27745,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":90,"thread_start_us":558728,"threads_started":2,"wal-append.queue_time_us":588242}
W20260929 14:12:40.160964 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 210474us
I20260929 14:12:40.178350 5242 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:40.179819 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 158669us
I20260929 14:12:40.094600 8395 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:12:40.140800 8459 log.cc:924] Time spent T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.168s user 0.000s sys 0.000s
I20260929 14:12:40.018534 8045 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.181895 5242 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:40.210148 8485 raft_consensus.cc:2726] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:40.146219 8368 consensus_queue.cc:237] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:40.190166 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 46722.2us
I20260929 14:12:40.190088 8395 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:40.128414 8375 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.187114 8456 raft_consensus.cc:2726] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:40.146839 8387 raft_consensus.cc:482] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:40.158685 8482 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:40.115365 8443 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:40.200582 5873 leader_election.cc:302] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:40.117735 8272 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:40.189751 8261 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:40.216645 8372 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:40.129083 8337 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:40.219213 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 199444us
W20260929 14:12:40.225023 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 245548us
I20260929 14:12:40.222918 5274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 2164) took 1165 ms. Trace:
I20260929 14:12:40.119171 8441 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.223804 8440 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:40.239243 5242 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:40.246672 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 274680us
W20260929 14:12:40.246459 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 45958.8us
I20260929 14:12:40.242868 5274 rpcz_store.cc:276] 0929 14:12:39.057980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:39.058123 (+ 143us) service_pool.cc:227] Handling call
0929 14:12:39.058499 (+ 376us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:39.058864 (+ 365us) raft_consensus.cc:1520] Early marking committed up to index 3
0929 14:12:39.081883 (+ 23019us) log.cc:842] Serialized 14 byte log entry
0929 14:12:39.536124 (+454241us) spinlock_profiling.cc:243] Waited 3.89 ms on lock 0x7b1800309580. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b45638 00007fbda1b48b11 00007fbda1b48ac9 00007fbda1b48a59 00007fbda1b48a21 00007fbda1b47d1d 00007fbdaada9204 00007fbdaada9039 00007fbda1b27dc6
0929 14:12:39.592777 (+ 56653us) spinlock_profiling.cc:243] Waited 1.65 ms on lock 0x7b1800309580. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda46fc82c 00007fbda1b279dc 00007fbdaa95ea39 00007fbda1b448bb 00007fbda1b42cbf 00007fbda1b44e22 00007fbda46ee5bb 00007fbda46f4063 00007fbda46f457b
0929 14:12:39.682316 (+ 89539us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:39.683217 (+ 901us) log.cc:842] Serialized 26 byte log entry
0929 14:12:39.683483 (+ 266us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:39.683545 (+ 62us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:39.683690 (+ 145us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:40.140248 (+456558us) spinlock_profiling.cc:243] Waited 4.48 ms on lock 0x7b44000232cc. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1a57a95 00007fbda1a584b2 00007fbda4700511 00007fbda46fbe1a 00007fbda46efbee 00007fbda46eef36 00007fbda46eeb58 00007fbda46f8551 00007fbda46f8509 00007fbda46f8499
0929 14:12:40.222739 (+ 82491us) raft_consensus.cc:1671] finished
0929 14:12:40.222760 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:40.222863 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19866,"prepare.queue_time_us":24826,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":98288,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":40926,"spinlock_wait_cycles":23047552,"thread_start_us":544764,"threads_started":2,"wal-append.queue_time_us":637959}
I20260929 14:12:40.247268 8501 raft_consensus.cc:2726] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:40.159222 8413 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:40.251933 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:40.252879 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:40.250437 5242 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:40.257261 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:40.259441 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:40.261158 5242 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:40.274628 5314 leader_election.cc:398] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 8. Current term is 9.
I20260929 14:12:40.236860 8387 raft_consensus.cc:504] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.241129 8272 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.386147 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:40.348117 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:40.459707 5314 leader_election.cc:401] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260929 14:12:40.341012 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:40.358121 5276 consensus_meta.cc:287] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.591s user 0.010s sys 0.001s
I20260929 14:12:40.455116 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:40.347476 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:40.253587 8261 raft_consensus.cc:2726] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260929 14:12:40.462070 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 61767.5us
I20260929 14:12:40.447738 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:40.463929 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 203000us
I20260929 14:12:40.462779 5276 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.469396 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:40.466240 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 191872us
I20260929 14:12:40.340502 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:40.247486 8474 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:40.294871 5197 raft_consensus.cc:1226] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:40.480377 5242 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:40.277261 8440 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:40.293648 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:40.468693 8480 raft_consensus.cc:2726] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:40.446367 8397 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:40.457100 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:40.508052 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:40.487924 5276 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:40.497365 5342 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:40.509258 5342 raft_consensus.cc:595] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:40.509622 5342 raft_consensus.cc:603] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:12:40.385989 8395 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:40.505570 8480 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260929 14:12:40.506316 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:40.512935 5343 raft_consensus.cc:3032] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Stepping down as leader of term 7
I20260929 14:12:40.513262 5343 raft_consensus.cc:729] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:40.507989 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:40.514324 5272 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:40.505136 5242 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:40.513962 5343 consensus_queue.cc:260] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.515076 5343 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:40.479409 8372 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473) with higher term. Message: Invalid argument: T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 8. Current term is 9.
I20260929 14:12:40.517795 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:40.551386 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:40.552141 5268 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 10.
I20260929 14:12:40.489342 8440 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:40.567162 5272 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:40.597076 5242 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:40.596565 5343 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
W20260929 14:12:40.601341 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 102072us
I20260929 14:12:40.602149 5274 raft_consensus.cc:1226] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:40.512614 8480 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:40.603828 5201 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
from {username='slave'} at 127.0.0.1:52182
I20260929 14:12:40.611341 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:40.613193 5242 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:40.614929 5343 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "599240a72b454fca841864f021dbbc64"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:40.567028 8395 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:40.616209 5343 raft_consensus.cc:595] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:40.559267 8045 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:40.596091 8440 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:40.620227 5239 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:40.622884 5242 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:40.618942 8502 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:40.239094 8482 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:40.635766 8504 raft_consensus.cc:482] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:40.636379 8504 raft_consensus.cc:504] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.639158 5343 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:40.639819 5343 raft_consensus.cc:595] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:40.516943 8397 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.651582 5242 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:40.613108 8415 consensus_meta.cc:287] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.611s user 0.009s sys 0.001s
I20260929 14:12:40.612695 8480 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:40.646559 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:40.623847 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:40.609556 8456 raft_consensus.cc:2726] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:40.646427 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:40.650565 8372 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:40.657528 5342 raft_consensus.cc:1261] T 62fed7cb04b44fca9312372e511f1c3e P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:40.618618 8454 raft_consensus.cc:2726] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:40.671460 5242 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:40.658584 8443 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:40.659073 8504 leader_election.cc:288] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:40.664462 5343 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:40.682243 5343 raft_consensus.cc:595] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:40.628509 8502 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:40.620649 8375 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:40.700011 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:40.675182 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:40.696692 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:40.697573 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:40.666028 8499 raft_consensus.cc:2781] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:40.706962 5242 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:40.670775 8338 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:40.711836 8397 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:40.714596 8509 log.cc:924] Time spent T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
W20260929 14:12:40.715416 8507 log.cc:924] Time spent T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260929 14:12:40.695096 8498 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260929 14:12:40.697415 8469 raft_consensus.cc:2726] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260929 14:12:40.717765 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 43884us
I20260929 14:12:40.766011 8498 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:40.766465 8498 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Advancing to term 11
I20260929 14:12:40.722232 8482 leader_election.cc:288] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:40.723421 8272 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:40.724538 8387 leader_election.cc:288] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:40.696404 8372 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:40.730604 8485 consensus_queue.cc:1035] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260929 14:12:40.732916 8395 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:40.750208 8261 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260929 14:12:40.665418 8452 raft_consensus.cc:2781] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:40.735632 8413 leader_election.cc:288] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:40.705032 8481 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:40.749380 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:40.751880 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:40.751905 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:12:40.757071 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:40.757537 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:40.804890 5269 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:40.806041 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:40.760303 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:40.762483 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:40.763394 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:40.699967 8337 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:40.755214 8338 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.738807 8499 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:40.734345 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:40.771107 8371 consensus_meta.cc:287] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.635s user 0.013s sys 0.006s
I20260929 14:12:40.659551 8415 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.831558 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:40.976895 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:40.860718 8499 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:40.986048 8415 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:40.789399 8261 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:40.987324 5348 raft_consensus.cc:1261] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260929 14:12:40.995033 8261 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:40.995227 8433 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:41.000965 8433 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.002983 8433 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:41.004025 8479 consensus_queue.cc:1035] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:41.005333 8375 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:41.006006 8375 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:41.007964 8375 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:40.831614 8395 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.013684 8395 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:41.011543 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:41.015087 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:41.014810 8261 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.017642 8372 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:41.020426 8261 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:41.028110 8480 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.036764 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:41.045984 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:40.974737 8338 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:41.051487 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:41.053391 8338 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:41.054124 8338 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:41.056833 8338 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:40.926218 8320 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:40.793635 8441 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:41.070039 5276 raft_consensus.cc:1226] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 9. Current term is 10. Ops: []
I20260929 14:12:41.071041 8320 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:40.969173 8502 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:40.831453 8452 raft_consensus.cc:686] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:40.830973 8481 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.408586 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:41.223603 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "678eb0db533d4bea872808b6fb39b764" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:41.096733 8460 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:41.223943 8480 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:41.410549 8460 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:41.413507 8460 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:41.130404 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:41.069653 8461 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
W20260929 14:12:41.134822 5165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.529s (SENT) [suppressed 5 similar messages]
W20260929 14:12:41.416028 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 296155us
I20260929 14:12:41.416822 8452 consensus_queue.cc:237] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.148663 8498 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.103766 8484 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:41.420028 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:12:41.420140 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 288995us
I20260929 14:12:41.421588 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:41.426280 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:41.529592 8484 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:41.531872 8484 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:41.427963 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:41.428231 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:41.428488 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:41.532806 5195 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:41.317061 8502 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:41.533351 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:41.533608 8502 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.534708 5197 raft_consensus.cc:1261] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260929 14:12:41.535574 8502 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:41.407186 8517 log.cc:924] Time spent T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.178s user 0.001s sys 0.000s
I20260929 14:12:41.431423 8441 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:41.536449 8441 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.536919 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:41.537447 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:41.538120 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:41.538777 8441 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:41.439224 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:41.541885 5344 raft_consensus.cc:1261] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260929 14:12:41.543728 8415 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:41.543915 8498 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:41.544288 8415 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.545568 8479 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:41.546049 8479 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:41.406281 8516 log.cc:924] Time spent T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.218s user 0.000s sys 0.000s
I20260929 14:12:41.450230 5192 raft_consensus.cc:1226] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:41.469993 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:41.470592 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:41.480314 8320 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:41.483409 8481 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:41.482928 8499 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.072988 8371 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:41.426910 8489 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:41.148118 8440 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.549258 8475 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:41.552063 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:41.554136 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:41.490552 8480 raft_consensus.cc:482] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:41.550496 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:41.565071 5194 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 5.
I20260929 14:12:41.565588 5345 raft_consensus.cc:1226] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:41.567899 5275 raft_consensus.cc:1226] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:41.551640 8461 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:41.564329 8489 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:41.580193 8479 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:41.585337 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:41.590463 8371 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:41.585337 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:41.586768 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:41.570374 8475 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:41.593001 5347 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47252 (request call id 2349) took 1229 ms (client timeout 1560 ms). Trace:
I20260929 14:12:41.585816 8415 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:41.577797 8480 raft_consensus.cc:504] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:41.585861 8440 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:41.594182 5347 rpcz_store.cc:269] 0929 14:12:40.364084 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:40.364198 (+ 114us) service_pool.cc:227] Handling call
0929 14:12:41.585187 (+1220989us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b5800183998. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:12:41.592927 (+ 7740us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4680960}
W20260929 14:12:41.613471 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 478649us
W20260929 14:12:41.618058 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 99950.7us
W20260929 14:12:41.626632 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 59889.2us
I20260929 14:12:41.627424 5201 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:41.627821 5201 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:41.637151 5316 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:41.638495 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:41.641858 5199 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:37270 (request call id 492) took 1050 ms. Trace:
I20260929 14:12:41.653508 5199 rpcz_store.cc:276] 0929 14:12:40.591961 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:40.632329 (+ 40368us) service_pool.cc:227] Handling call
0929 14:12:41.641247 (+1008918us) spinlock_profiling.cc:243] Waited 1.1 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:12:41.641427 (+ 180us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:41.641581 (+ 154us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:41.641607 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:41.641644 (+ 37us) raft_consensus.cc:1631] Marking committed up to 8
0929 14:12:41.641679 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:41.641724 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:41.641806 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2522880}
W20260929 14:12:41.635685 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 72572.1us
I20260929 14:12:41.646575 8475 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:41.647867 8480 leader_election.cc:288] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:41.648952 8461 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:41.650174 8489 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:41.634900 8499 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:41.658854 8337 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:41.661969 8337 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:41.662384 8337 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
W20260929 14:12:41.662897 8508 log.cc:924] Time spent T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 1.068s user 0.012s sys 0.016s
I20260929 14:12:41.664214 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:41.665006 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:41.666972 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:41.667626 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:41.674063 5196 raft_consensus.cc:1226] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260929 14:12:41.674868 5196 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 2180) took 1117 ms. Trace:
I20260929 14:12:41.675472 8524 consensus_queue.cc:1035] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:12:41.675251 5196 rpcz_store.cc:276] 0929 14:12:40.557101 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:40.557551 (+ 450us) service_pool.cc:227] Handling call
0929 14:12:41.673543 (+1115992us) spinlock_profiling.cc:243] Waited 938 us on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:12:41.673763 (+ 220us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:41.674587 (+ 824us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:41.674806 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2156288}
I20260929 14:12:41.680009 8455 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:41.680609 8455 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.681550 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:41.680157 5164 leader_election.cc:302] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:41.683014 8455 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:41.685012 8395 consensus_queue.cc:1035] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:41.685130 5269 raft_consensus.cc:1226] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 9. Current term is 10. Ops: []
I20260929 14:12:41.686152 8489 raft_consensus.cc:2726] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:41.678597 5316 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:41.688009 8524 raft_consensus.cc:2781] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:41.688359 8524 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:41.688766 8524 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:41.691807 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:41.695439 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:41.696547 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:41.692907 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:41.699067 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:41.698087 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:41.700948 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:41.655630 5201 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.697151 5194 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:41.701691 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:41.711898 5316 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:41.713270 5316 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:41.714600 5316 leader_election.cc:302] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:41.714952 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:41.711889 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:41.716696 5314 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:41.716989 5199 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
W20260929 14:12:41.719642 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 83565.8us
I20260929 14:12:41.720141 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:41.722482 5871 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:41.733839 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:41.716084 8372 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:41.737896 5201 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:41.747740 5314 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:41.750826 5314 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:41.757619 5201 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:52182 (request call id 2178) took 1182 ms. Trace:
I20260929 14:12:41.758087 5201 rpcz_store.cc:276] 0929 14:12:40.556598 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:40.557609 (+ 1011us) service_pool.cc:227] Handling call
0929 14:12:41.627229 (+1069620us) spinlock_profiling.cc:243] Waited 1.15 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c6030 00007fbda99b13f9 00007fbda99b0e95 00007fbda30a2784
0929 14:12:41.738726 (+111497us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2648960}
I20260929 14:12:41.724116 8502 raft_consensus.cc:2726] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:41.724323 8441 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:41.822530 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:41.729548 8496 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:41.825342 5349 raft_consensus.cc:3032] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Stepping down as leader of term 8
I20260929 14:12:41.829854 5349 raft_consensus.cc:729] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
W20260929 14:12:41.828097 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 101440us
I20260929 14:12:41.752367 8479 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:41.797302 8415 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:41.737041 8338 consensus_queue.cc:1046] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.101s
I20260929 14:12:41.798743 8370 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:41.823786 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
W20260929 14:12:41.866508 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 115777us
I20260929 14:12:41.834089 8526 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260929 14:12:41.881953 5314 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:12:41.859136 8372 raft_consensus.cc:2781] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260929 14:12:41.883234 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 149071us
I20260929 14:12:41.860740 8441 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260929 14:12:41.907858 5314 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:41.905252 8534 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:41.906988 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:41.835598 8496 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.694332 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260929 14:12:41.921893 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 197237us
I20260929 14:12:41.878762 8536 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:41.942207 8338 raft_consensus.cc:2781] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:41.879210 8415 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.882938 8370 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.943099 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:41.930045 8441 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:41.948742 5201 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
W20260929 14:12:41.919273 8508 log.cc:924] Time spent T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260929 14:12:41.925693 5099 catalog_manager.cc:5495] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 10 to 12, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 12 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } } }
I20260929 14:12:41.860198 8479 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.946154 8534 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:41.907477 8372 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:41.945307 8371 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260929 14:12:41.966167 8338 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:12:42.001586 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 77365.7us
I20260929 14:12:42.055354 5314 leader_election.cc:302] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:42.008622 5276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 2193) took 1259 ms. Trace:
I20260929 14:12:42.056520 5314 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:42.010360 5272 raft_consensus.cc:1226] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:42.009514 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.058547 5274 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50776 (request call id 2197) took 1122 ms. Trace:
I20260929 14:12:42.016841 5270 raft_consensus.cc:1261] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260929 14:12:42.058985 5274 rpcz_store.cc:276] 0929 14:12:40.936364 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:42.009257 (+1072893us) service_pool.cc:227] Handling call
0929 14:12:42.058470 (+ 49213us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:42.021926 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.000430 5873 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
W20260929 14:12:41.998742 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 282504us
I20260929 14:12:42.056267 5276 rpcz_store.cc:276] 0929 14:12:40.748909 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:40.749074 (+ 165us) service_pool.cc:227] Handling call
0929 14:12:40.749346 (+ 272us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:41.072502 (+323156us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:42.008413 (+935911us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x7b54003746d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:42.008570 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":737105152}
I20260929 14:12:42.019328 5346 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 2363) took 1362 ms. Trace:
I20260929 14:12:42.063514 5316 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:42.045578 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 115823us
I20260929 14:12:42.065416 5268 raft_consensus.cc:2379] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 10: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:42.063213 5346 rpcz_store.cc:276] 0929 14:12:40.656879 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:40.657917 (+ 1038us) service_pool.cc:227] Handling call
0929 14:12:40.658178 (+ 261us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:40.749809 (+ 91631us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:40.749852 (+ 43us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:40.749904 (+ 52us) raft_consensus.cc:1631] Marking committed up to 6
0929 14:12:40.749941 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:42.019126 (+1269185us) spinlock_profiling.cc:243] Waited 91.4 ms on lock 0x7b54003a9dd4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:42.019171 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:42.019275 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":210289024}
W20260929 14:12:42.068672 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 68337.8us
W20260929 14:12:42.008142 8527 log.cc:924] Time spent T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20260929 14:12:42.070669 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:42.073132 5268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 2146) took 1136 ms. Trace:
I20260929 14:12:42.076634 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.074703 5316 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:42.080798 5268 rpcz_store.cc:276] 0929 14:12:40.936940 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:42.021669 (+1084729us) service_pool.cc:227] Handling call
0929 14:12:42.073057 (+ 51388us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260929 14:12:42.160521 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:42.061148 8537 raft_consensus.cc:2726] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:42.205921 8537 consensus_queue.cc:1046] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260929 14:12:42.014410 8370 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:42.205952 5351 raft_consensus.cc:2379] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 8: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:42.168197 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.209764 5276 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:42.003432 8441 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Advancing to term 11
I20260929 14:12:42.006031 8372 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:42.009554 8415 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:42.010260 8338 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:42.222141 8479 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:42.067605 8545 raft_consensus.cc:2781] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:42.080212 8536 raft_consensus.cc:2781] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:42.249389 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.250252 5276 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 8.
W20260929 14:12:42.251677 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 182293us
I20260929 14:12:42.245293 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.254240 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.252616 5314 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:42.080274 8546 raft_consensus.cc:2781] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:42.012203 8524 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.258834 5268 raft_consensus.cc:2379] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 10: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
W20260929 14:12:42.246698 8337 consensus_meta.cc:287] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.584s user 0.013s sys 0.006s
I20260929 14:12:42.271075 8337 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:42.261755 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 187126us
I20260929 14:12:42.265662 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.251128 8536 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:42.275144 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:42.281683 5201 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:42.282655 5200 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 2179) took 1725 ms. Trace:
I20260929 14:12:42.286556 5200 rpcz_store.cc:276] 0929 14:12:40.556834 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:40.556949 (+ 115us) service_pool.cc:227] Handling call
0929 14:12:42.281827 (+1724878us) spinlock_profiling.cc:243] Waited 1.32 ms on lock 0x7b5800000f98. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99c32b0 00007fbda99ae7f9 00007fbda99ae2a2 00007fbda3094684
0929 14:12:42.282071 (+ 244us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:42.282276 (+ 205us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:42.282311 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:42.282362 (+ 51us) raft_consensus.cc:1631] Marking committed up to 3
0929 14:12:42.282405 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:42.282469 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:42.282587 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3041536}
I20260929 14:12:42.288475 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.302476 5276 raft_consensus.cc:1261] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260929 14:12:42.300988 5314 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:42.304152 5268 raft_consensus.cc:3037] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Advancing to term 11
I20260929 14:12:42.247424 8545 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:42.257455 8546 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:42.271606 8535 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:42.293426 8371 raft_consensus.cc:979] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:42.273144 8337 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:42.310307 8546 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:42.309856 8545 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:42.311232 8543 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:42.312223 8539 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:42.316049 5349 consensus_queue.cc:260] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.305598 8548 raft_consensus.cc:2726] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:42.350492 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:42.299108 8452 raft_consensus.cc:465] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Not starting pre-election -- already a leader
I20260929 14:12:42.322611 5268 raft_consensus.cc:1261] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260929 14:12:42.327273 5201 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:42.288542 8560 raft_consensus.cc:979] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:42.336606 5345 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 529) took 1906 ms. Trace:
I20260929 14:12:42.340811 8533 raft_consensus.cc:3032] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Stepping down as leader of term 7
I20260929 14:12:42.276788 8540 raft_consensus.cc:2781] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:42.299337 8440 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:42.289065 8536 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:42.348867 8454 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:42.376391 8548 raft_consensus.cc:2726] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:42.378525 5345 rpcz_store.cc:276] 0929 14:12:40.430532 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:40.470349 (+ 39817us) service_pool.cc:227] Handling call
0929 14:12:40.470662 (+ 313us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:41.570092 (+1099430us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:42.336376 (+766284us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540027e054. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a6df51 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772 00007fbda2e9794c 00007fbda2e96d86
0929 14:12:42.336544 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2517699328}
I20260929 14:12:42.380515 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.401671 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.402879 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:42.400931 5871 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.570s user 0.005s sys 0.008s
W20260929 14:12:42.410945 5871 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.580s user 0.005s sys 0.008s
W20260929 14:12:42.414542 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 584015us
W20260929 14:12:42.405892 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 82205.2us
I20260929 14:12:42.379769 8533 raft_consensus.cc:729] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:42.399782 8540 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:42.411226 8560 consensus_queue.cc:1035] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.021s
I20260929 14:12:42.434166 8560 raft_consensus.cc:1067] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 LEADER]: Signalling peer c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:42.435454 8524 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:42.436447 8563 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:42.437537 8562 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:42.439426 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:42.440181 5345 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:42.353986 8543 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.413268 8534 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:42.377920 8337 consensus_queue.cc:1046] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260929 14:12:42.502302 5164 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:42.451969 5871 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:42.504961 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.452915 5349 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
W20260929 14:12:42.462975 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 117719us
I20260929 14:12:42.420266 8548 raft_consensus.cc:1067] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Signalling peer c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:42.376745 8338 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:42.353821 8539 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.409025 8440 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.477110 5347 raft_consensus.cc:1226] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:42.492542 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:42.494419 5343 raft_consensus.cc:1261] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260929 14:12:42.496125 5342 raft_consensus.cc:1226] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:42.514222 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.499820 8501 consensus_queue.cc:1046] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260929 14:12:42.509056 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.505847 5270 raft_consensus.cc:2454] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
W20260929 14:12:42.517895 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 91143.8us
I20260929 14:12:42.519325 5871 leader_election.cc:302] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:42.426218 8540 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:42.414027 8454 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.532511 8454 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:42.349530 8535 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.533537 5276 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:42.534569 8535 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:42.515836 5267 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
W20260929 14:12:42.524745 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 68233.7us
I20260929 14:12:42.545418 5272 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "599240a72b454fca841864f021dbbc64"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:50776
I20260929 14:12:42.545686 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:42.546139 5272 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:42.546511 5272 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Advancing to term 11
I20260929 14:12:42.404109 8556 consensus_queue.cc:1046] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.106s
I20260929 14:12:42.547107 8533 consensus_queue.cc:260] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.547487 5269 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:50776
I20260929 14:12:42.548197 5269 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:42.548614 5269 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:42.438078 8562 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.549382 8539 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:42.550814 8562 leader_election.cc:288] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:42.551549 8461 raft_consensus.cc:2726] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:42.543383 5276 raft_consensus.cc:2396] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260929 14:12:42.513283 8551 raft_consensus.cc:2781] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:42.555006 8372 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.555647 8551 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:42.556041 8551 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:42.556173 5348 raft_consensus.cc:1226] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:42.558334 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:42.560060 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:42.560827 8440 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:42.437007 8563 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:42.567600 8563 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:42.569479 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.571682 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.572058 8496 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:42.567067 5269 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.569981 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.570286 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:42.526355 8560 consensus_queue.cc:1046] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 14:12:42.530104 8441 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.573186 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.702095 5273 raft_consensus.cc:2379] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 9: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:42.612172 5242 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
W20260929 14:12:42.704473 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 92409.5us
W20260929 14:12:42.605849 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 82333.3us
I20260929 14:12:42.705735 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.601707 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.707194 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:42.708163 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 189380us
I20260929 14:12:42.586256 5349 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 9.
I20260929 14:12:42.702390 5275 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:42.738957 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:42.739360 5164 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:42.740260 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.741695 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.742098 5096 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52750 (request call id 224) took 7366 ms. Trace:
I20260929 14:12:42.743397 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:42.744604 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:42.745570 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:42.718775 5272 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.738073 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:42.777878 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:42.755573 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:42.557060 8372 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:42.781998 5351 raft_consensus.cc:1226] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 10. Current term is 11. Ops: []
I20260929 14:12:42.698493 8524 raft_consensus.cc:3032] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Stepping down as leader of term 8
I20260929 14:12:42.747150 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e50b67ada64545a6b7525d4aba317bb0" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:42.749680 5269 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:42.784188 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.748287 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:42.777105 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 64469.6us
I20260929 14:12:42.747153 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.747879 5198 raft_consensus.cc:1226] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:42.777529 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:42.748670 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.763298 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:12:42.784543 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 43833.4us
I20260929 14:12:42.784614 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:42.790427 5195 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:42.784787 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:42.785755 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62fed7cb04b44fca9312372e511f1c3e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:42.762876 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:42.787048 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:42.747977 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.789927 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.746500 5342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:42.610910 8565 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:42.805555 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.792861 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.807865 8441 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 11 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:42.808422 5276 raft_consensus.cc:1226] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 10. Current term is 11. Ops: []
I20260929 14:12:42.809422 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64181b973d25469da29a63928ede710f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:42.793095 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.794298 5903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
W20260929 14:12:42.794301 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 85534.4us
W20260929 14:12:42.777024 8554 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 9 current term: 9 is higher.
I20260929 14:12:42.796183 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:42.779969 5201 raft_consensus.cc:1261] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260929 14:12:42.797333 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:42.818914 5194 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:42.616050 8543 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:42.799494 5271 raft_consensus.cc:2379] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 10: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:42.781605 8455 consensus_queue.cc:1046] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260929 14:12:42.715516 8561 raft_consensus.cc:2726] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:42.804535 8537 raft_consensus.cc:3032] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 LEADER]: Stepping down as leader of term 9
I20260929 14:12:42.829461 8537 raft_consensus.cc:729] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:42.830338 8537 consensus_queue.cc:260] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.833545 5193 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52182 (request call id 2196) took 1134 ms. Trace:
I20260929 14:12:42.804710 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:42.833927 5193 rpcz_store.cc:276] 0929 14:12:41.699473 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:41.699725 (+ 252us) service_pool.cc:227] Handling call
0929 14:12:41.700007 (+ 282us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:41.700298 (+ 291us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:41.700328 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:42.432790 (+732462us) spinlock_profiling.cc:243] Waited 6.41 ms on lock 0x7b180017bba0. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda1b12f96 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda3450e11 00007fbda3460442 00007fbda4aa8c11 00007fbda4a6d58a
0929 14:12:42.469937 (+ 37147us) spinlock_profiling.cc:243] Waited 1.5 ms on lock 0x7b2c0005ac28. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda3450e11 00007fbda3460442 00007fbda4aa8c11 00007fbda4a6d58a 00007fbda4a6c99e 00007fbda99ae2e9 00007fbda3094684 00007fbda3094612 00007fbda3094541 00007fbda30944bc 00007fbda3093772
0929 14:12:42.470534 (+ 597us) log.cc:842] Serialized 26 byte log entry
0929 14:12:42.776683 (+306149us) raft_consensus.cc:1631] Marking committed up to 4
0929 14:12:42.778042 (+ 1359us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:42.778108 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:42.833339 (+ 55231us) raft_consensus.cc:1671] finished
0929 14:12:42.833368 (+ 29us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:42.833492 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":100,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":9745,"spinlock_wait_cycles":18206336,"thread_start_us":305776,"threads_started":1,"wal-append.queue_time_us":358605}
I20260929 14:12:42.834856 8440 consensus_queue.cc:1035] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:12:42.805493 8565 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.836645 8455 consensus_queue.cc:1046] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 14:12:42.837191 8565 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:42.625356 8475 raft_consensus.cc:2726] T 520249ed5be343dcb30c86d3396491a4 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:42.785169 8539 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:42.742427 5096 rpcz_store.cc:276] 0929 14:12:35.375407 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:35.375638 (+ 231us) service_pool.cc:227] Handling call
0929 14:12:42.742042 (+7366404us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:12:39.229610 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:12:39.230101 (+ 491us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:12:39.230135 (+ 34us) write_op.cc:435] Acquired schema lock
0929 14:12:39.230151 (+ 16us) tablet.cc:660] Decoding operations
0929 14:12:39.231465 (+ 1314us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:12:39.231522 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0929 14:12:39.231549 (+ 27us) tablet.cc:683] Acquiring locks for 17 operations
0929 14:12:39.232715 (+ 1166us) tablet.cc:699] Row locks acquired
0929 14:12:39.232737 (+ 22us) write_op.cc:260] PREPARE: finished
0929 14:12:39.232898 (+ 161us) write_op.cc:270] Start()
0929 14:12:39.233019 (+ 121us) write_op.cc:276] Timestamp: P: 1790691159232844 usec, L: 0
0929 14:12:39.233040 (+ 21us) op_driver.cc:331] REPLICATION: starting
0929 14:12:39.256256 (+ 23216us) log.cc:842] Serialized 6699 byte log entry
0929 14:12:40.465809 (+1209553us) op_driver.cc:444] REPLICATION: finished
0929 14:12:41.071037 (+605228us) write_op.cc:301] APPLY: starting
0929 14:12:41.071128 (+ 91us) tablet.cc:1365] starting BulkCheckPresence
0929 14:12:41.071448 (+ 320us) tablet.cc:1368] finished BulkCheckPresence
0929 14:12:41.071596 (+ 148us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:12:41.470529 (+398933us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:12:41.470843 (+ 314us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=17
0929 14:12:41.470901 (+ 58us) write_op.cc:312] APPLY: finished
0929 14:12:41.471289 (+ 388us) log.cc:842] Serialized 119 byte log entry
0929 14:12:41.628348 (+157059us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:41.637936 (+ 9588us) write_op.cc:454] Released schema lock
0929 14:12:41.651542 (+ 13606us) write_op.cc:341] FINISH: Updating metrics
0929 14:12:41.657877 (+ 6335us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:41.657921 (+ 44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1601572,"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":563206,"apply.run_cpu_time_us":9568,"apply.run_wall_time_us":588059,"num_ops":17,"prepare.queue_time_us":1351893,"prepare.run_cpu_time_us":53113,"prepare.run_wall_time_us":448724,"raft.queue_time_us":226970,"raft.run_cpu_time_us":82232,"raft.run_wall_time_us":472778,"replication_time_us":1226200,"thread_start_us":1532493,"threads_started":5,"wal-append.queue_time_us":774987,"wal-append.run_cpu_time_us":20769,"wal-append.run_wall_time_us":1561478}]]}
I20260929 14:12:42.795221 5898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5acb0315ba44b5a9d145119868e7168" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:42.526394 8338 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:42.812727 5274 raft_consensus.cc:2379] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 9: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:42.812975 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:42.867192 5269 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Advancing to term 11
I20260929 14:12:42.747601 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.814339 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.815122 5272 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:42.664007 8566 raft_consensus.cc:3032] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 LEADER]: Stepping down as leader of term 9
I20260929 14:12:42.825452 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b630b7184c5456bad1fbd45514e52ba" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:42.828694 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:42.786412 8524 raft_consensus.cc:729] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:42.675225 8545 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.670799 8540 raft_consensus.cc:504] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.781666 8496 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election won for term 9
I20260929 14:12:42.671226 8551 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.853153 5343 raft_consensus.cc:1226] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 5. Current term is 6. Ops: []
I20260929 14:12:42.854322 5194 raft_consensus.cc:2454] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 10.
I20260929 14:12:42.851764 5195 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:42.855337 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.857666 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:42.739974 8533 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:42.864849 5899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:42.879105 8499 consensus_queue.cc:1046] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:42.880789 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:42.911103 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:42.911912 5351 raft_consensus.cc:3032] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:42.630599 8546 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:42.887764 5269 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 11.
I20260929 14:12:42.700340 8536 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:42.897312 5316 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.545s user 0.021s sys 0.014s
W20260929 14:12:42.926152 5316 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.574s user 0.021s sys 0.014s
I20260929 14:12:42.908433 5242 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:42.912262 5351 raft_consensus.cc:729] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:42.912777 5345 raft_consensus.cc:1226] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:42.918928 5314 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:42.920948 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:42.931766 5344 raft_consensus.cc:3032] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 LEADER]: Stepping down as leader of term 10
I20260929 14:12:42.932117 5344 raft_consensus.cc:729] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:42.921294 8496 raft_consensus.cc:686] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:42.920948 8499 raft_consensus.cc:3032] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:42.881403 8566 raft_consensus.cc:729] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:42.932931 5344 consensus_queue.cc:260] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.938767 5344 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Advancing to term 11
I20260929 14:12:42.940047 8440 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:42.940901 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:42.864156 8535 consensus_queue.cc:1046] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260929 14:12:42.942258 8535 raft_consensus.cc:3032] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:42.942579 8486 raft_consensus.cc:3032] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 8 LEADER]: Stepping down as leader of term 8
I20260929 14:12:42.943012 8486 raft_consensus.cc:729] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:42.943440 8565 raft_consensus.cc:2781] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Leader election won for term 11
I20260929 14:12:42.943935 8565 raft_consensus.cc:686] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:42.943804 8486 consensus_queue.cc:260] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:42.944890 8565 consensus_queue.cc:237] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.946465 8566 consensus_queue.cc:260] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.947680 8566 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:42.937345 8494 consensus_queue.cc:1046] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260929 14:12:42.930229 5351 consensus_queue.cc:260] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.954370 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:42.932494 5316 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:42.955494 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:42.956982 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e230699b3fb4d4d97fdb04988babce5" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:42.961428 5348 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:42.962635 8496 consensus_queue.cc:237] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.867183 8558 consensus_queue.cc:1046] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.072s
I20260929 14:12:42.937819 8499 raft_consensus.cc:729] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:42.966823 8499 consensus_queue.cc:260] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.968016 8499 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:42.971099 8551 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:42.972745 8540 leader_election.cc:288] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:42.974215 8545 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:42.974968 8524 consensus_queue.cc:260] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.903407 8481 raft_consensus.cc:3032] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 10 LEADER]: Stepping down as leader of term 10
I20260929 14:12:42.976265 8481 raft_consensus.cc:729] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 10 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:42.976619 8441 raft_consensus.cc:2726] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260929 14:12:42.977200 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.977068 8481 consensus_queue.cc:260] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.978338 8481 raft_consensus.cc:3037] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Advancing to term 11
I20260929 14:12:42.980618 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:42.980708 8546 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:42.942596 8535 raft_consensus.cc:729] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:42.982493 8536 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:42.982261 8535 consensus_queue.cc:260] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.873098 8539 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:42.983476 8535 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:42.984632 8420 raft_consensus.cc:3032] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Stepping down as leader of term 7
I20260929 14:12:42.984866 8539 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:42.935702 5346 raft_consensus.cc:1226] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:42.936232 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:42.977530 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 45147.6us
I20260929 14:12:42.986053 5271 raft_consensus.cc:2379] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 10: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:42.936440 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.989105 5345 raft_consensus.cc:1226] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:42.991377 5196 raft_consensus.cc:1261] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260929 14:12:42.992594 5344 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 11.
I20260929 14:12:42.997844 8494 raft_consensus.cc:3032] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 5 LEADER]: Stepping down as leader of term 5
I20260929 14:12:42.998256 8494 raft_consensus.cc:729] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:43.000321 5198 raft_consensus.cc:1261] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260929 14:12:43.000909 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:43.003100 5274 raft_consensus.cc:1226] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:43.003942 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:42.985046 8420 raft_consensus.cc:729] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 7 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:43.009712 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:43.002043 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:42.945148 8486 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:43.013091 5316 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.018729 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:43.019579 5200 raft_consensus.cc:3037] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:43.019316 8494 consensus_queue.cc:260] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:43.020371 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:43.022935 5316 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.024228 8533 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:43.024676 8533 raft_consensus.cc:686] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:43.025893 5906 raft_consensus.cc:1226] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:43.025537 8533 consensus_queue.cc:237] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.027292 8420 consensus_queue.cc:260] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.028638 8420 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:43.001288 8440 consensus_queue.cc:1046] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.028s
I20260929 14:12:43.029889 8545 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260929 14:12:43.030670 8537 raft_consensus.cc:3037] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:43.032207 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:43.013422 5351 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:43.034583 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:43.035559 5196 raft_consensus.cc:2379] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 8: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:43.050096 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:43.065766 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e61c0e0d757e43f19868f329ee0b8bec" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:43.071578 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:43.076581 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:43.077596 5199 raft_consensus.cc:2454] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 3.
I20260929 14:12:43.095201 5199 raft_consensus.cc:1226] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:43.109576 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:43.109824 5200 raft_consensus.cc:2454] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 10.
I20260929 14:12:43.110227 5199 raft_consensus.cc:3037] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Advancing to term 11
I20260929 14:12:43.113583 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:43.112768 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:43.114184 5195 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:43.116271 5198 raft_consensus.cc:3037] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:43.071380 8562 consensus_queue.cc:1046] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260929 14:12:43.119786 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:12:43.120337 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 227299us
I20260929 14:12:43.125554 5201 raft_consensus.cc:1226] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 9. Current term is 10. Ops: []
I20260929 14:12:43.121953 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:43.128108 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:43.123036 5351 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 7.
I20260929 14:12:43.134553 5239 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:43.139531 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:43.154419 5201 raft_consensus.cc:1226] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 9. Current term is 10. Ops: []
I20260929 14:12:43.156720 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "321633daf3914178a03ee03e5c0159d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:43.157569 5201 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:43.154920 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:43.159040 5239 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:43.160384 5239 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:43.184090 5201 raft_consensus.cc:2454] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:43.157860 8480 raft_consensus.cc:2781] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Leader election won for term 11
I20260929 14:12:43.193542 5268 raft_consensus.cc:1226] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 10. Current term is 11. Ops: []
I20260929 14:12:43.199479 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:43.166128 8543 raft_consensus.cc:2741] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
W20260929 14:12:43.206713 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 337410us
I20260929 14:12:43.209326 8337 consensus_queue.cc:1046] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260929 14:12:43.215241 8575 log.cc:924] Time spent T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.059s user 0.000s sys 0.000s
I20260929 14:12:43.193208 8563 raft_consensus.cc:2781] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:43.227154 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:43.165980 8440 consensus_queue.cc:1046] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260929 14:12:43.237934 5242 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.239270 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:43.233590 8525 consensus_queue.cc:1046] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260929 14:12:43.248618 8543 raft_consensus.cc:3032] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Stepping down as leader of term 8
I20260929 14:12:43.249042 8543 raft_consensus.cc:729] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:43.249858 8543 consensus_queue.cc:260] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.250955 8543 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:43.252774 8554 raft_consensus.cc:3032] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 LEADER]: Stepping down as leader of term 10
I20260929 14:12:43.252915 8563 raft_consensus.cc:686] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:43.253261 8554 raft_consensus.cc:729] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:43.253858 8563 consensus_queue.cc:237] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.254055 8554 consensus_queue.cc:260] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.255149 8554 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Advancing to term 11
I20260929 14:12:43.259939 8494 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:43.267237 5196 raft_consensus.cc:2454] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 11.
I20260929 14:12:43.250970 8524 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
W20260929 14:12:43.290743 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 52890.5us
I20260929 14:12:43.291738 5242 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.293095 5242 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.237960 8452 raft_consensus.cc:3032] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:43.293825 8452 raft_consensus.cc:729] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:43.294387 5242 leader_election.cc:302] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.294587 8452 consensus_queue.cc:260] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:43.295459 5242 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:43.295861 8452 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:43.296774 8544 raft_consensus.cc:3032] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:43.297142 8544 raft_consensus.cc:729] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:43.297685 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:43.297899 8544 consensus_queue.cc:260] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.299306 8544 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:43.301452 8498 raft_consensus.cc:2726] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:43.302685 5242 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:43.303197 5316 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.303830 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:43.308063 5199 raft_consensus.cc:2454] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 11.
I20260929 14:12:43.308643 5195 raft_consensus.cc:1226] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 10. Current term is 11. Ops: []
I20260929 14:12:43.304391 5194 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
W20260929 14:12:43.317463 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 126718us
I20260929 14:12:43.319229 5316 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.302012 8498 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:43.317688 8454 raft_consensus.cc:2726] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:43.327471 5316 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.330547 5242 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.332435 5316 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.326215 8372 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:43.317685 8440 raft_consensus.cc:2726] T e50b67ada64545a6b7525d4aba317bb0 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:43.337502 5242 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:43.325774 8535 raft_consensus.cc:2726] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:43.342947 5316 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.344542 5316 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.345683 5316 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.347250 8572 raft_consensus.cc:2726] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260929 14:12:43.347503 8337 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260929 14:12:43.336030 8395 raft_consensus.cc:2726] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 LEADER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:43.333601 8480 raft_consensus.cc:686] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:43.354524 5242 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.335196 8539 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:43.328572 8568 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260929 14:12:43.339320 8498 raft_consensus.cc:2726] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:43.350247 8387 raft_consensus.cc:3032] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 LEADER]: Stepping down as leader of term 7
I20260929 14:12:43.370591 5242 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:43.342360 8372 raft_consensus.cc:482] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:43.350001 8545 raft_consensus.cc:2781] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:43.363054 8454 raft_consensus.cc:2726] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:43.381467 8577 consensus_queue.cc:1035] T f2610284ac4749c5bb443c221df8ff92 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:12:43.394630 5242 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.397769 5193 raft_consensus.cc:2454] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:43.378212 5198 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 7.
I20260929 14:12:43.382804 5165 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.391666 8537 raft_consensus.cc:2781] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:43.391609 8545 raft_consensus.cc:686] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:43.378346 8497 consensus_queue.cc:1035] T 465f57e1a1654ed1b40ce88c27fa52ac P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.023s
I20260929 14:12:43.401234 8545 consensus_queue.cc:237] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.400229 8562 raft_consensus.cc:2726] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:43.399323 5200 raft_consensus.cc:1226] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:43.407308 5165 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.408551 5165 leader_election.cc:302] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.409413 8481 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:43.409693 8320 raft_consensus.cc:2726] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:43.410043 8481 raft_consensus.cc:2726] T e5acb0315ba44b5a9d145119868e7168 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:43.410324 8586 raft_consensus.cc:2726] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260929 14:12:43.412726 5165 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.413849 5242 leader_election.cc:302] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.386097 8372 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:43.415915 5165 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.379083 8387 raft_consensus.cc:729] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:43.422477 5165 leader_election.cc:302] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.382812 8499 consensus_queue.cc:1035] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260929 14:12:43.423853 5242 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.425158 5165 leader_election.cc:302] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.429611 8455 raft_consensus.cc:2781] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:43.432821 5165 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.421077 8587 raft_consensus.cc:2726] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:43.438304 8440 raft_consensus.cc:2781] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader election won for term 10
I20260929 14:12:43.438868 8440 raft_consensus.cc:686] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:43.419499 8497 raft_consensus.cc:2781] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:43.432125 8541 raft_consensus.cc:2726] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:43.434439 5242 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.442404 5165 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.439798 8440 consensus_queue.cc:237] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.435760 8455 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:43.444494 8537 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:43.444844 8577 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:43.445006 8537 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:43.445289 8577 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:43.445672 8577 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:43.444868 8455 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:43.449431 8525 raft_consensus.cc:2726] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260929 14:12:43.440177 8420 raft_consensus.cc:2726] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:43.430030 8480 consensus_queue.cc:237] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.443535 8497 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:43.461112 8497 raft_consensus.cc:3037] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:43.463706 8387 consensus_queue.cc:260] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.464673 5165 leader_election.cc:302] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.436188 8567 raft_consensus.cc:2726] T 7b630b7184c5456bad1fbd45514e52ba P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:43.465639 8420 raft_consensus.cc:2726] T 64181b973d25469da29a63928ede710f P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:43.466634 5165 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.467434 8420 raft_consensus.cc:2726] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:43.467782 5165 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.464816 8387 raft_consensus.cc:3037] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:43.469439 8420 raft_consensus.cc:2726] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:43.474342 8372 raft_consensus.cc:504] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.474380 8577 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.482719 8577 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:43.481426 8455 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:43.483747 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 49561.3us
I20260929 14:12:43.484436 5200 raft_consensus.cc:1226] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 6. Current term is 7. Ops: []
I20260929 14:12:43.484943 5242 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.486454 5242 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:43.487452 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:43.491583 8497 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.491979 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:43.493907 5165 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.496049 5165 leader_election.cc:302] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.496618 8577 raft_consensus.cc:2726] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:43.496964 8455 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:43.498518 5200 raft_consensus.cc:1261] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:43.500633 8577 raft_consensus.cc:2726] T 5e230699b3fb4d4d97fdb04988babce5 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:43.501492 5165 leader_election.cc:302] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.502681 8525 raft_consensus.cc:2726] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:43.506624 5347 raft_consensus.cc:1226] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:43.508598 5165 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.509464 8455 raft_consensus.cc:2726] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:43.510705 5165 leader_election.cc:302] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.511705 5165 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.513904 8525 raft_consensus.cc:2726] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:43.515313 8455 raft_consensus.cc:2726] T 4300e83384e9458bb280a4b1e3131932 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:43.503969 5316 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.520879 8545 raft_consensus.cc:2781] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:43.521571 8545 raft_consensus.cc:686] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:43.522487 8545 consensus_queue.cc:237] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.528853 8387 raft_consensus.cc:2726] T 5e230699b3fb4d4d97fdb04988babce5 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:43.528790 8499 raft_consensus.cc:2726] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260929 14:12:43.528898 8567 raft_consensus.cc:2726] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260929 14:12:43.530728 8337 consensus_queue.cc:1046] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:12:43.531077 8539 consensus_queue.cc:1035] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:12:43.532014 5165 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.532347 8338 raft_consensus.cc:3032] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:43.532804 8338 raft_consensus.cc:729] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:43.533631 8338 consensus_queue.cc:260] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.534691 8338 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:43.536945 5165 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.537753 8420 raft_consensus.cc:2726] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260929 14:12:43.538197 5165 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.539009 8420 raft_consensus.cc:2726] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 14:12:43.539805 5165 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.540107 5268 raft_consensus.cc:1261] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:43.540652 8541 raft_consensus.cc:2726] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:43.545498 8554 consensus_queue.cc:1035] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:43.541049 5165 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
W20260929 14:12:43.555413 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 69064us
I20260929 14:12:43.556427 8541 raft_consensus.cc:2726] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:43.556842 8455 raft_consensus.cc:2726] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:43.557857 5165 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.558612 8544 raft_consensus.cc:2726] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260929 14:12:43.563086 5242 leader_election.cc:302] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.563932 8567 raft_consensus.cc:2726] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260929 14:12:43.567749 8372 leader_election.cc:288] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:43.568358 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:43.570480 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:43.571130 5193 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:43.571790 5276 raft_consensus.cc:1261] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260929 14:12:43.564798 8480 consensus_queue.cc:1035] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260929 14:12:43.564939 5242 leader_election.cc:302] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.578068 8480 raft_consensus.cc:2726] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260929 14:12:43.578593 8337 consensus_queue.cc:1035] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 14:12:43.578931 5200 raft_consensus.cc:1261] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260929 14:12:43.579082 5242 leader_election.cc:302] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.580322 8567 raft_consensus.cc:2726] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260929 14:12:43.581974 8372 consensus_queue.cc:1035] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 14:12:43.528743 8537 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.590078 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:43.591686 8497 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:43.592636 5242 leader_election.cc:302] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.594063 8440 raft_consensus.cc:2726] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:43.594199 5242 leader_election.cc:302] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.595594 5242 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:12:43.596443 8440 raft_consensus.cc:2726] T e61c0e0d757e43f19868f329ee0b8bec P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:43.597126 8567 raft_consensus.cc:2781] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:43.597558 8567 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:43.597915 8567 raft_consensus.cc:3037] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:43.603004 8537 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:43.604846 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:43.605531 8440 consensus_queue.cc:1046] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 14:12:43.606544 8452 raft_consensus.cc:3032] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 LEADER]: Stepping down as leader of term 7
I20260929 14:12:43.606900 8452 raft_consensus.cc:729] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:43.607602 8452 consensus_queue.cc:260] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.608701 8452 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:43.612560 5242 leader_election.cc:302] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.613571 8440 raft_consensus.cc:2781] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader election won for term 10
I20260929 14:12:43.614074 8440 raft_consensus.cc:686] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:43.614971 8440 consensus_queue.cc:237] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.619424 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9ebe16909c64202bcf70eb0425efdca" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:43.620077 5347 raft_consensus.cc:1226] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:43.620597 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d2589e399004a6797ef987deb148b6c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:43.619946 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:43.634613 8567 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:43.639070 8567 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:43.690428 5099 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:44250 (request call id 131) took 5409 ms. Trace:
W20260929 14:12:43.695361 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 137425us
I20260929 14:12:43.691262 5099 rpcz_store.cc:276] 0929 14:12:38.280773 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:38.280927 (+ 154us) service_pool.cc:227] Handling call
0929 14:12:43.690149 (+5409222us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:12:43.122768 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:12:43.257349 (+134581us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:12:43.257386 (+ 37us) write_op.cc:435] Acquired schema lock
0929 14:12:43.257403 (+ 17us) tablet.cc:660] Decoding operations
0929 14:12:43.257736 (+ 333us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:12:43.257810 (+ 74us) write_op.cc:641] Partition lock acquired for write op
0929 14:12:43.257835 (+ 25us) tablet.cc:683] Acquiring locks for 1 operations
0929 14:12:43.258016 (+ 181us) tablet.cc:699] Row locks acquired
0929 14:12:43.258033 (+ 17us) write_op.cc:260] PREPARE: finished
0929 14:12:43.258164 (+ 131us) write_op.cc:270] Start()
0929 14:12:43.258263 (+ 99us) write_op.cc:276] Timestamp: P: 1790691163258126 usec, L: 0
0929 14:12:43.258305 (+ 42us) op_driver.cc:331] REPLICATION: starting
0929 14:12:43.258868 (+ 563us) log.cc:842] Serialized 533 byte log entry
0929 14:12:43.595366 (+336498us) op_driver.cc:444] REPLICATION: finished
0929 14:12:43.635850 (+ 40484us) write_op.cc:301] APPLY: starting
0929 14:12:43.635932 (+ 82us) tablet.cc:1365] starting BulkCheckPresence
0929 14:12:43.636025 (+ 93us) tablet.cc:1368] finished BulkCheckPresence
0929 14:12:43.636211 (+ 186us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:12:43.636467 (+ 256us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:12:43.636640 (+ 173us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0929 14:12:43.636681 (+ 41us) write_op.cc:312] APPLY: finished
0929 14:12:43.636869 (+ 188us) log.cc:842] Serialized 23 byte log entry
0929 14:12:43.637021 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:43.637106 (+ 85us) write_op.cc:454] Released schema lock
0929 14:12:43.637147 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0929 14:12:43.637451 (+ 304us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:43.637509 (+ 58us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":937867,"reported_tablets":1,"child_traces":[["op",{"apply.queue_time_us":40207,"apply.run_cpu_time_us":1862,"apply.run_wall_time_us":1897,"num_ops":1,"prepare.queue_time_us":387503,"prepare.run_cpu_time_us":5688,"prepare.run_wall_time_us":236159,"raft.queue_time_us":142911,"raft.run_cpu_time_us":19084,"raft.run_wall_time_us":40405,"replication_time_us":336975,"thread_start_us":553392,"threads_started":4,"wal-append.queue_time_us":109389}]]}
I20260929 14:12:43.701488 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:43.694672 5098 catalog_manager.cc:5495] T 7b630b7184c5456bad1fbd45514e52ba P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 6 to 7, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 7 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:12:43.703172 5200 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
W20260929 14:12:43.705766 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 85423.4us
I20260929 14:12:43.710763 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:43.718263 5165 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.732200 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:43.735549 8387 consensus_queue.cc:1046] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:43.738983 8420 raft_consensus.cc:2726] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:43.743338 8454 raft_consensus.cc:3032] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Stepping down as leader of term 8
I20260929 14:12:43.749449 8454 raft_consensus.cc:729] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:43.750978 8454 consensus_queue.cc:260] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.752847 8454 raft_consensus.cc:3037] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:43.757752 5873 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.763864 8462 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260929 14:12:43.764313 5242 leader_election.cc:302] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:43.765585 8452 raft_consensus.cc:2741] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260929 14:12:43.751020 5098 catalog_manager.cc:5495] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:43.767141 5347 raft_consensus.cc:1226] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:43.772755 8440 consensus_queue.cc:1046] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260929 14:12:43.773983 8440 raft_consensus.cc:3032] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Stepping down as leader of term 8
I20260929 14:12:43.774406 8440 raft_consensus.cc:729] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:43.775458 8440 consensus_queue.cc:260] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.776960 8440 raft_consensus.cc:3037] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:43.803434 5198 raft_consensus.cc:1226] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 7. Current term is 8. Ops: []
I20260929 14:12:43.808307 8545 consensus_queue.cc:1046] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260929 14:12:43.811012 8545 raft_consensus.cc:3032] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Stepping down as leader of term 7
I20260929 14:12:43.811403 8545 raft_consensus.cc:729] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:43.811896 8587 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:43.809572 5269 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 8.
I20260929 14:12:43.827751 5193 raft_consensus.cc:2454] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 9.
I20260929 14:12:43.834280 5316 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.836153 8554 raft_consensus.cc:2781] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election won for term 9
I20260929 14:12:43.812592 8545 consensus_queue.cc:260] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.836719 8554 raft_consensus.cc:686] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:43.837762 8545 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:43.839466 8554 consensus_queue.cc:237] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.812603 8587 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.846664 8587 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:43.847157 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:43.861008 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:43.864565 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:43.867439 5272 raft_consensus.cc:1261] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260929 14:12:43.870662 5193 raft_consensus.cc:1226] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Rejecting Update request from peer d886fb1eb0474d2d9f7410c4013c45e4 for earlier term 4. Current term is 5. Ops: []
I20260929 14:12:43.869097 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:43.877099 5193 raft_consensus.cc:1261] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260929 14:12:43.876940 8539 consensus_queue.cc:1046] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:12:43.878633 8537 raft_consensus.cc:3032] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260929 14:12:43.879154 8537 raft_consensus.cc:729] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:43.880375 8537 consensus_queue.cc:260] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.882584 8537 raft_consensus.cc:3037] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:43.888191 8539 consensus_queue.cc:1035] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:43.889668 5193 raft_consensus.cc:1226] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:43.849052 5200 raft_consensus.cc:2454] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 9.
I20260929 14:12:43.908020 5196 raft_consensus.cc:1226] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260929 14:12:43.910130 5351 raft_consensus.cc:1261] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260929 14:12:43.910374 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
W20260929 14:12:43.901294 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 52008.5us
I20260929 14:12:43.926470 5196 raft_consensus.cc:1261] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260929 14:12:43.943738 5316 leader_election.cc:302] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:43.937701 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:43.957477 8433 consensus_queue.cc:1035] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260929 14:12:43.961812 5194 raft_consensus.cc:2379] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 8: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:43.975426 5268 raft_consensus.cc:1226] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 3. Current term is 4. Ops: []
I20260929 14:12:43.955485 8539 raft_consensus.cc:2781] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election won for term 9
I20260929 14:12:43.977628 8539 raft_consensus.cc:686] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:43.978737 8539 consensus_queue.cc:237] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:43.985210 5346 raft_consensus.cc:1261] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260929 14:12:44.021615 8580 consensus_queue.cc:1046] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260929 14:12:44.022333 5350 raft_consensus.cc:1261] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260929 14:12:44.022511 8543 consensus_queue.cc:1035] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
I20260929 14:12:44.022908 8580 raft_consensus.cc:3032] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Stepping down as leader of term 3
I20260929 14:12:44.023344 8580 raft_consensus.cc:729] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:44.024350 8580 consensus_queue.cc:260] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:44.026070 5165 leader_election.cc:302] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:44.028400 5165 leader_election.cc:302] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:44.028849 8496 raft_consensus.cc:2726] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:44.029589 8496 raft_consensus.cc:2726] T 6d2589e399004a6797ef987deb148b6c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260929 14:12:44.041528 5096 catalog_manager.cc:5495] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 6 to 7, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 7 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:12:44.045137 5271 raft_consensus.cc:1261] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260929 14:12:44.046878 8537 consensus_queue.cc:1035] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260929 14:12:44.044076 8589 consensus_queue.cc:1035] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:44.044353 5346 raft_consensus.cc:1261] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260929 14:12:44.052423 8543 consensus_queue.cc:1035] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260929 14:12:44.046804 5096 catalog_manager.cc:5495] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:44.057965 5165 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:44.043485 5269 raft_consensus.cc:3037] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:44.047863 8580 raft_consensus.cc:3037] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:44.059051 5096 catalog_manager.cc:5495] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 7 to 8, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 8 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:12:44.061848 8577 raft_consensus.cc:2726] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260929 14:12:44.061262 5096 catalog_manager.cc:5495] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 4 to 7, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 7 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:44.062980 5096 catalog_manager.cc:5495] T 62fed7cb04b44fca9312372e511f1c3e P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 5 to 6, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 6 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:44.068922 5098 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52726 (request call id 228) took 8284 ms. Trace:
I20260929 14:12:44.069335 5098 rpcz_store.cc:276] 0929 14:12:35.784767 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:35.784978 (+ 211us) service_pool.cc:227] Handling call
0929 14:12:44.068858 (+8283880us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:12:43.798959 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:12:43.799714 (+ 755us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:12:43.804469 (+ 4755us) write_op.cc:435] Acquired schema lock
0929 14:12:43.806453 (+ 1984us) tablet.cc:660] Decoding operations
0929 14:12:43.828538 (+ 22085us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:12:43.828641 (+ 103us) write_op.cc:641] Partition lock acquired for write op
0929 14:12:43.828674 (+ 33us) tablet.cc:683] Acquiring locks for 2 operations
0929 14:12:43.829158 (+ 484us) tablet.cc:699] Row locks acquired
0929 14:12:43.829191 (+ 33us) write_op.cc:260] PREPARE: finished
0929 14:12:43.829388 (+ 197us) write_op.cc:270] Start()
0929 14:12:43.829570 (+ 182us) write_op.cc:276] Timestamp: P: 1790691163829314 usec, L: 0
0929 14:12:43.829609 (+ 39us) op_driver.cc:331] REPLICATION: starting
0929 14:12:43.830280 (+ 671us) log.cc:842] Serialized 923 byte log entry
0929 14:12:43.858176 (+ 27896us) op_driver.cc:444] REPLICATION: finished
0929 14:12:43.859086 (+ 910us) write_op.cc:301] APPLY: starting
0929 14:12:43.859220 (+ 134us) tablet.cc:1365] starting BulkCheckPresence
0929 14:12:43.859406 (+ 186us) tablet.cc:1368] finished BulkCheckPresence
0929 14:12:43.859571 (+ 165us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:12:43.860091 (+ 520us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:12:43.860411 (+ 320us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0929 14:12:43.860462 (+ 51us) write_op.cc:312] APPLY: finished
0929 14:12:43.860776 (+ 314us) log.cc:842] Serialized 29 byte log entry
0929 14:12:43.860955 (+ 179us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:43.861036 (+ 81us) write_op.cc:454] Released schema lock
0929 14:12:43.861083 (+ 47us) write_op.cc:341] FINISH: Updating metrics
0929 14:12:43.861539 (+ 456us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:43.861568 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":867618,"reported_tablets":52,"child_traces":[["op",{"apply.queue_time_us":529,"apply.run_cpu_time_us":2643,"apply.run_wall_time_us":2783,"num_ops":2,"prepare.queue_time_us":25122,"prepare.run_cpu_time_us":3762,"prepare.run_wall_time_us":37033,"replication_time_us":28445}]]}
I20260929 14:12:44.076766 5343 raft_consensus.cc:1261] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260929 14:12:44.069098 5269 raft_consensus.cc:1261] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260929 14:12:44.064955 5096 catalog_manager.cc:5495] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:44.090971 8589 consensus_queue.cc:1035] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260929 14:12:44.092684 8554 consensus_queue.cc:1035] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.002s
I20260929 14:12:44.116256 5165 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:44.135289 8577 raft_consensus.cc:2726] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260929 14:12:44.134382 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 53865.2us
I20260929 14:12:44.139693 5096 catalog_manager.cc:5495] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 4 to 5, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 5 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:44.146193 5196 raft_consensus.cc:1261] T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260929 14:12:44.148034 5165 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:44.150947 8539 consensus_queue.cc:1035] T a39f12a08e914559b1a12cbac3e3cec4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:44.153460 8433 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260929 14:12:44.147559 8577 consensus_queue.cc:1035] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
W20260929 14:12:44.156610 8611 log.cc:924] Time spent T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20260929 14:12:44.158417 5350 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:44.159178 5350 raft_consensus.cc:595] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:44.164518 5348 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:44.165197 5348 raft_consensus.cc:595] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:44.165596 5348 raft_consensus.cc:603] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:12:44.170266 5165 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:44.149436 5096 catalog_manager.cc:5495] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 6 to 7, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 7 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:44.174078 8496 raft_consensus.cc:2781] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:44.174734 8496 raft_consensus.cc:686] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [term 8 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:44.175738 8496 consensus_queue.cc:237] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:44.224766 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 54628.8us
I20260929 14:12:44.227900 5097 catalog_manager.cc:5495] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 7 to 11, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 11 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:44.230285 5097 catalog_manager.cc:5495] T 879dc5894e734d098eb2d6064e10712a P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 10 to 11, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 11 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:44.234355 5097 catalog_manager.cc:5495] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 5 to 8. New cstate: current_term: 8 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:44.244146 5193 raft_consensus.cc:1261] T f9ebe16909c64202bcf70eb0425efdca P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260929 14:12:44.239751 5097 catalog_manager.cc:5495] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:44.245675 8545 consensus_queue.cc:1035] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:12:44.245818 5242 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:44.248847 5097 catalog_manager.cc:5495] T 05a81ea24145409b82fe949d849ab73d P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 6 to 7, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 7 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:44.252849 5097 catalog_manager.cc:5495] T 321633daf3914178a03ee03e5c0159d9 P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 7 to 8, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 8 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:44.257990 8454 raft_consensus.cc:2726] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:12:44.262606 5194 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:12:44.263295 5194 raft_consensus.cc:595] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:44.270161 5198 raft_consensus.cc:1261] T 599240a72b454fca841864f021dbbc64 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260929 14:12:44.270740 5195 raft_consensus.cc:1261] T 22fb3fae8e7947989abe31ce58000d58 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260929 14:12:44.271288 5194 raft_consensus.cc:1261] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260929 14:12:44.279377 5351 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
mode: GRACEFUL
new_leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:44238
I20260929 14:12:44.280131 5351 raft_consensus.cc:595] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Received request to transfer leadership to TS c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:44.280541 5351 raft_consensus.cc:603] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260929 14:12:44.281360 8454 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:44.282012 8454 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.284349 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260929 14:12:44.285180 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:44.285907 5350 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:44.286598 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:12:44.286953 5239 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:44.287573 8454 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:44.287796 8440 raft_consensus.cc:2781] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:44.288177 8440 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:44.288517 8440 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:44.289407 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:44.296308 8440 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.300516 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:44.301354 5350 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
W20260929 14:12:44.305503 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 59778us
I20260929 14:12:44.306521 8454 consensus_queue.cc:1035] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 14:12:44.306883 5242 leader_election.cc:302] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:44.309085 5096 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52736 (request call id 226) took 1356 ms. Trace:
I20260929 14:12:44.309482 5096 rpcz_store.cc:276] 0929 14:12:42.952406 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:42.955649 (+ 3243us) service_pool.cc:227] Handling call
0929 14:12:44.308995 (+1353346us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:12:44.200862 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:12:44.201406 (+ 544us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:12:44.201455 (+ 49us) write_op.cc:435] Acquired schema lock
0929 14:12:44.201480 (+ 25us) tablet.cc:660] Decoding operations
0929 14:12:44.206554 (+ 5074us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:12:44.206622 (+ 68us) write_op.cc:641] Partition lock acquired for write op
0929 14:12:44.206653 (+ 31us) tablet.cc:683] Acquiring locks for 8 operations
0929 14:12:44.207400 (+ 747us) tablet.cc:699] Row locks acquired
0929 14:12:44.207420 (+ 20us) write_op.cc:260] PREPARE: finished
0929 14:12:44.207548 (+ 128us) write_op.cc:270] Start()
0929 14:12:44.207650 (+ 102us) write_op.cc:276] Timestamp: P: 1790691164207506 usec, L: 0
0929 14:12:44.207670 (+ 20us) op_driver.cc:331] REPLICATION: starting
0929 14:12:44.210310 (+ 2640us) log.cc:842] Serialized 3234 byte log entry
0929 14:12:44.215020 (+ 4710us) op_driver.cc:444] REPLICATION: finished
0929 14:12:44.217770 (+ 2750us) write_op.cc:301] APPLY: starting
0929 14:12:44.217848 (+ 78us) tablet.cc:1365] starting BulkCheckPresence
0929 14:12:44.218037 (+ 189us) tablet.cc:1368] finished BulkCheckPresence
0929 14:12:44.218145 (+ 108us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:12:44.219066 (+ 921us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:12:44.219246 (+ 180us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0929 14:12:44.219289 (+ 43us) write_op.cc:312] APPLY: finished
0929 14:12:44.219520 (+ 231us) log.cc:842] Serialized 65 byte log entry
0929 14:12:44.219660 (+ 140us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:44.219772 (+ 112us) write_op.cc:454] Released schema lock
0929 14:12:44.219814 (+ 42us) write_op.cc:341] FINISH: Updating metrics
0929 14:12:44.220047 (+ 233us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:44.220072 (+ 25us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":109972,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":1172,"apply.run_cpu_time_us":2618,"apply.run_wall_time_us":2619,"num_ops":8,"prepare.queue_time_us":100,"prepare.run_cpu_time_us":7711,"prepare.run_wall_time_us":9701,"replication_time_us":7252}]]}
I20260929 14:12:44.315006 8440 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:44.318810 5271 raft_consensus.cc:1261] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260929 14:12:44.314532 8454 raft_consensus.cc:2726] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260929 14:12:44.327926 8619 log.cc:924] Time spent T a39f12a08e914559b1a12cbac3e3cec4 P ac17d3b553b74c9da39e0ab307addacd: Append to log took a long time: real 0.065s user 0.001s sys 0.000s
I20260929 14:12:44.329347 8440 consensus_queue.cc:1035] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 14:12:44.329720 8499 consensus_queue.cc:1035] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260929 14:12:44.330714 8440 consensus_queue.cc:1035] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20260929 14:12:44.337873 8337 consensus_queue.cc:1035] T f9ebe16909c64202bcf70eb0425efdca P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 14:12:44.341289 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:44.367758 5350 raft_consensus.cc:2454] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 7.
I20260929 14:12:44.352717 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:44.393563 8543 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:44.415205 5271 raft_consensus.cc:1261] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260929 14:12:44.441754 5239 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:44.456948 8537 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:44.467964 8537 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.471140 8537 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:44.463061 8499 raft_consensus.cc:2781] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:44.472124 8499 raft_consensus.cc:686] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:44.445631 8337 consensus_queue.cc:1035] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260929 14:12:44.476854 8525 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:44.494585 8525 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.496704 8525 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:44.497834 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:44.498733 5350 raft_consensus.cc:2354] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7.
I20260929 14:12:44.500144 5164 leader_election.cc:398] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7.
I20260929 14:12:44.500473 5164 leader_election.cc:401] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:12:44.501236 8496 raft_consensus.cc:3037] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 7
I20260929 14:12:44.477843 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:44.483261 8580 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:44.475167 8528 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:44.429124 8543 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:44.542135 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:44.545006 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:44.546878 5268 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 9.
I20260929 14:12:44.556669 5198 raft_consensus.cc:1261] T 7e9a6f1c02ae4024813fda5af5c665da P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260929 14:12:44.556957 8499 consensus_queue.cc:237] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.513968 8580 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:44.558598 8537 consensus_queue.cc:1035] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.015s
I20260929 14:12:44.585340 8543 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:44.542095 8528 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:44.587591 8528 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:44.587500 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:44.588515 5901 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:44.589529 8580 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:44.589988 5242 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:44.590708 8627 raft_consensus.cc:979] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd: : Instructing follower c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:44.590478 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:44.591187 8627 raft_consensus.cc:1067] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Signalling peer c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:44.591295 5268 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:12:44.592782 5871 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:44.593848 8580 raft_consensus.cc:2781] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:44.594395 8580 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:44.594866 8580 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:44.594872 8543 raft_consensus.cc:2781] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:44.597707 8543 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:44.598237 8543 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:44.598667 8496 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 6. Current term is 7.
I20260929 14:12:44.601838 5314 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:44.611869 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:44.609596 8337 raft_consensus.cc:2781] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260929 14:12:44.613188 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:44.616205 8337 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:44.616376 5343 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 9.
I20260929 14:12:44.613397 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:44.613600 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:44.616619 8337 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:44.622347 5346 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 6.
I20260929 14:12:44.622417 8626 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:44.623342 5350 raft_consensus.cc:2379] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 6: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:44.623051 8626 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.625695 8626 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:44.626724 8626 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:44.626873 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:44.627650 5269 raft_consensus.cc:2379] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 11: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:44.627239 8626 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.629161 8626 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:44.639082 8337 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.640964 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:44.641743 5272 raft_consensus.cc:2379] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 9: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
I20260929 14:12:44.644366 8539 raft_consensus.cc:979] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4: : Instructing follower c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:44.644878 8539 raft_consensus.cc:1067] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Signalling peer c5ec116e39144881bdf1f2cd3e923cf5 to start an election
I20260929 14:12:44.645761 5268 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:50770
I20260929 14:12:44.646517 5268 raft_consensus.cc:482] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:44.646940 5268 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:44.648777 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:44.649462 5195 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 10
I20260929 14:12:44.654870 5272 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982"
dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
from {username='slave'} at 127.0.0.1:50776
I20260929 14:12:44.655494 5272 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:44.655930 5272 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:44.663610 8543 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:44.675863 5870 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:44.677341 8488 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260929 14:12:44.677798 8488 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:44.678220 8488 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:44.682230 5097 catalog_manager.cc:5495] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 8 to 9, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 9 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:44.685633 5097 catalog_manager.cc:5495] T 599240a72b454fca841864f021dbbc64 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 10 to 11, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 11 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:44.687048 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:44.700065 8543 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:44.702314 8337 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:44.702140 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:44.703423 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:44.710659 5269 raft_consensus.cc:3037] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:44.706022 5097 catalog_manager.cc:5495] T 22fb3fae8e7947989abe31ce58000d58 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 9 to 10, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 10 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:44.707285 5347 raft_consensus.cc:2379] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 6: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:44.709260 5195 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 10.
I20260929 14:12:44.712180 5097 catalog_manager.cc:5495] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 6 to 7, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 7 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:44.713411 5097 catalog_manager.cc:5495] T f9ebe16909c64202bcf70eb0425efdca P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 8 to 9, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 9 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:44.706220 5268 raft_consensus.cc:504] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.714769 5097 catalog_manager.cc:5495] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 7 to 8, leader changed from ac17d3b553b74c9da39e0ab307addacd (127.3.242.65) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 8 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } }
I20260929 14:12:44.716379 5097 catalog_manager.cc:5495] T 625b1b3f659041f8b54de3499628ebc4 P c5ec116e39144881bdf1f2cd3e923cf5 reported cstate change: term changed from 9 to 10, leader changed from d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67) to c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66). New cstate: current_term: 10 leader_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:44.722890 5316 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:44.723209 8580 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:44.724426 5906 raft_consensus.cc:2354] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:44.724457 8537 raft_consensus.cc:2781] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader election won for term 10
I20260929 14:12:44.725097 8537 raft_consensus.cc:686] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:44.725992 8537 consensus_queue.cc:237] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.727762 5242 leader_election.cc:398] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Vote denied by peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497) with higher term. Message: Invalid argument: T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:44.728170 5242 leader_election.cc:401] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260929 14:12:44.728914 8535 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:44.732514 8554 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:44.733040 8554 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.733500 8496 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:44.734040 8496 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:44.735145 8554 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:44.737636 8496 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:44.737066 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:44.738711 5274 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:44.739127 8580 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:44.740646 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:44.742107 5314 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:44.742976 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:44.743525 5350 raft_consensus.cc:3037] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:44.744649 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:44.745334 5194 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:44.753255 5350 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 7.
I20260929 14:12:44.754160 5269 raft_consensus.cc:2454] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 10.
I20260929 14:12:44.754362 5870 leader_election.cc:302] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:44.755522 8478 raft_consensus.cc:2781] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:44.755774 8554 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:44.757712 8554 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:44.758184 8554 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:44.764905 5276 raft_consensus.cc:1261] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260929 14:12:44.767800 5268 leader_election.cc:288] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:44.768756 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:44.769415 5343 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:44.707508 5272 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.771009 8539 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:44.771620 8539 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.772380 5272 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:44.754204 8488 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:44.774528 8539 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:44.775096 8488 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 10 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:44.775245 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:44.777420 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:44.778060 5347 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:44.756070 8478 raft_consensus.cc:686] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 7 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:44.780179 8478 consensus_queue.cc:237] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:44.786636 5350 raft_consensus.cc:3032] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Stepping down as leader of term 8
I20260929 14:12:44.786502 8433 consensus_queue.cc:1035] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:12:44.787184 5350 raft_consensus.cc:729] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:44.788110 5350 consensus_queue.cc:260] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.791729 5350 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:44.792693 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:44.789307 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:44.797664 5198 raft_consensus.cc:3037] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 10
I20260929 14:12:44.799700 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cd4876c229b4b768fe4180899274e70" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:44.800365 5193 raft_consensus.cc:3032] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Stepping down as leader of term 6
I20260929 14:12:44.800699 5193 raft_consensus.cc:729] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:44.807257 5343 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 7.
I20260929 14:12:44.803341 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:44.809777 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:44.810750 5195 raft_consensus.cc:3037] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:44.808321 5347 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 10.
I20260929 14:12:44.814674 5901 raft_consensus.cc:2354] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 9. Current term is 10.
I20260929 14:12:44.820489 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:44.824631 5239 leader_election.cc:302] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:44.827351 5350 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 9.
I20260929 14:12:44.826022 5870 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:44.830883 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:44.831578 5239 leader_election.cc:302] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:44.837258 8440 raft_consensus.cc:2781] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election won for term 7
I20260929 14:12:44.837546 8454 raft_consensus.cc:2781] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election won for term 9
I20260929 14:12:44.829792 5193 consensus_queue.cc:260] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.845085 5343 raft_consensus.cc:3037] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:44.839272 5167 leader_election.cc:398] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Vote denied by peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497) with higher term. Message: Invalid argument: T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 9. Current term is 10.
I20260929 14:12:44.847543 5167 leader_election.cc:401] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:12:44.848333 5096 catalog_manager.cc:5495] T 7e9a6f1c02ae4024813fda5af5c665da P d886fb1eb0474d2d9f7410c4013c45e4 reported cstate change: term changed from 8 to 9, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67). New cstate: current_term: 9 leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: HEALTHY } } }
I20260929 14:12:44.855665 5351 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 11.
I20260929 14:12:44.820586 8535 raft_consensus.cc:2726] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497) with higher term. Message: Invalid argument: T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 6. Current term is 7.
I20260929 14:12:44.876801 5193 raft_consensus.cc:3037] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Advancing to term 7
W20260929 14:12:44.878537 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 48777.6us
I20260929 14:12:44.851647 8440 raft_consensus.cc:686] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:44.890501 8501 raft_consensus.cc:2781] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Leader election won for term 10
I20260929 14:12:44.890532 5274 raft_consensus.cc:2454] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 7.
I20260929 14:12:44.890995 5195 raft_consensus.cc:2454] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 9.
I20260929 14:12:44.822718 8554 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.909595 5343 raft_consensus.cc:1261] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260929 14:12:44.913467 5198 raft_consensus.cc:2454] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 10.
I20260929 14:12:44.918917 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:44.919988 5345 raft_consensus.cc:2354] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 9. Current term is 10.
I20260929 14:12:44.920850 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:44.921797 5199 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:44.923064 5316 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:44.924446 8537 raft_consensus.cc:2781] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:44.924868 8537 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:44.925156 8501 raft_consensus.cc:686] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 10 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:44.925292 8537 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:44.926470 8501 consensus_queue.cc:237] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:44.930394 8440 consensus_queue.cc:237] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.948002 5193 raft_consensus.cc:2454] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 7.
W20260929 14:12:44.946408 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 64008.6us
I20260929 14:12:44.917614 8626 raft_consensus.cc:2726] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497) with higher term. Message: Invalid argument: T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 9. Current term is 10.
I20260929 14:12:44.949710 8454 raft_consensus.cc:686] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:44.962966 8640 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:44.963538 8640 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.958712 8589 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:44.940976 8587 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:44.965593 8640 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:44.982292 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:44.983309 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 87158.7us
I20260929 14:12:44.965049 8589 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:44.990262 5345 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 9.
I20260929 14:12:44.996502 5239 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:44.965620 8587 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:45.003544 5097 catalog_manager.cc:5495] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "675b77a0a88a4bf08134281fc0cd949e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260929 14:12:44.998685 5164 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:12:45.073462 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 190819us
W20260929 14:12:45.074010 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 75415.7us
I20260929 14:12:45.074728 5164 leader_election.cc:302] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:45.083501 8537 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:45.085807 8537 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:45.087445 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:45.088093 5193 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:45.095760 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:45.096485 5198 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:45.101711 5193 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 9.
I20260929 14:12:45.103029 5316 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:45.088519 8433 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260929 14:12:45.091959 8592 log.cc:924] Time spent T 00000000000000000000000000000000 P 101c95b7e4d04f699d0fa198361f4d29: Append to log took a long time: real 0.100s user 0.002s sys 0.000s
I20260929 14:12:45.075554 8539 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:45.013690 8454 consensus_queue.cc:237] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:45.110390 8337 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election won for term 9
I20260929 14:12:45.110903 8337 raft_consensus.cc:686] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:45.111713 8337 consensus_queue.cc:237] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:45.091820 8543 raft_consensus.cc:482] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:45.126060 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 51443.8us
I20260929 14:12:45.129402 8625 raft_consensus.cc:2726] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:45.131930 8647 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:45.131942 8645 consensus_queue.cc:1035] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:12:45.107609 8554 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:45.104061 8433 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:45.110289 8639 raft_consensus.cc:2781] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260929 14:12:45.149034 8589 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:45.150900 8639 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:45.116883 8539 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:45.164870 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:45.166741 5901 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:45.168429 5242 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:45.124651 8543 raft_consensus.cc:504] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:45.154842 8647 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:45.171907 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:45.172801 5345 raft_consensus.cc:2454] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:45.156230 8433 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Advancing to term 11
I20260929 14:12:45.162056 5198 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:45.154105 8587 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:45.169235 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 82346.3us
I20260929 14:12:45.174868 5343 raft_consensus.cc:1261] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260929 14:12:45.182140 8647 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:45.183799 8543 leader_election.cc:288] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:45.196985 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:45.166085 8639 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:45.197676 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:45.199579 5345 raft_consensus.cc:2354] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 7. Current term is 8.
I20260929 14:12:45.199718 5343 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:45.201072 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:45.206647 5271 raft_consensus.cc:3037] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 8
I20260929 14:12:45.210441 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:45.211285 5272 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Advancing to term 9
W20260929 14:12:45.219247 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 50871.9us
I20260929 14:12:45.222268 5272 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 9.
I20260929 14:12:45.227291 5239 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:45.238251 5239 leader_election.cc:398] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 7. Current term is 8.
I20260929 14:12:45.238729 5239 leader_election.cc:401] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260929 14:12:45.243242 5239 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:45.244262 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 130767us
I20260929 14:12:45.235102 8647 raft_consensus.cc:2781] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:45.251108 5316 leader_election.cc:302] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:45.238786 8640 raft_consensus.cc:2781] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260929 14:12:45.253757 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 129462us
I20260929 14:12:45.260012 8539 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:45.262027 8554 raft_consensus.cc:2781] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:45.262516 8554 raft_consensus.cc:686] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:45.263403 8554 consensus_queue.cc:237] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:45.264942 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:45.265610 5199 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:45.267722 8454 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:45.268142 8454 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:45.268280 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:45.269248 5193 raft_consensus.cc:2379] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 5: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:45.268560 8454 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:45.271442 5268 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:45.274552 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 146821us
I20260929 14:12:45.273249 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:45.275287 5164 leader_election.cc:398] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 9. Current term is 10.
I20260929 14:12:45.275442 5195 raft_consensus.cc:2379] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 8: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:45.285295 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbca949bc56f4fb19a19b93566383d8d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:45.241336 8499 consensus_queue.cc:1035] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
I20260929 14:12:45.253342 8647 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:45.300276 5272 raft_consensus.cc:1261] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260929 14:12:45.273239 8640 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:45.292141 5198 raft_consensus.cc:2354] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 7. Current term is 8.
I20260929 14:12:45.286589 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:45.350781 5199 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
I20260929 14:12:45.354693 8650 raft_consensus.cc:482] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
W20260929 14:12:45.355942 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 104910us
I20260929 14:12:45.320410 8525 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:45.349095 8640 raft_consensus.cc:3037] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:45.352988 8647 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:45.388423 5198 raft_consensus.cc:1261] T 5e18ad4fc5524f4687d35e8b2dd03193 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260929 14:12:45.355171 8650 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:45.399585 8650 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:45.382465 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 139321us
I20260929 14:12:45.295727 8433 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:45.384311 5195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:45.405864 8537 consensus_queue.cc:1035] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260929 14:12:45.381524 8525 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:45.467438 8554 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 14:12:45.508291 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 165072us
I20260929 14:12:45.515110 5242 leader_election.cc:398] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Vote denied by peer ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147) with higher term. Message: Invalid argument: T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 7. Current term is 8.
W20260929 14:12:45.522231 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 119357us
I20260929 14:12:45.524653 5316 leader_election.cc:302] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:12:45.547292 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 136384us
I20260929 14:12:45.548647 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:45.549804 5901 raft_consensus.cc:2379] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 5: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:12:45.519280 8554 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:45.531950 8433 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:45.562067 5193 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260929 14:12:45.542039 8648 raft_consensus.cc:465] T 929c62a6106e42d08a13689e62dce3b1 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 LEADER]: Not starting pre-election -- already a leader
I20260929 14:12:45.584180 8643 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260929 14:12:45.588650 8525 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:45.591872 5193 raft_consensus.cc:1261] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260929 14:12:45.608875 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 82732.5us
I20260929 14:12:45.618880 8537 consensus_queue.cc:1035] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:12:45.624325 5198 raft_consensus.cc:1261] T 2282d5a97d5a49b9a71bc53d08a7cb65 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260929 14:12:45.599469 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 78882.9us
I20260929 14:12:45.556674 8494 consensus_queue.cc:1035] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.222s
I20260929 14:12:45.641956 8663 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:45.637737 8674 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:45.655901 8454 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:45.658445 8640 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:45.659680 8672 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:45.661067 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:45.662962 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:45.665422 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:45.667615 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:45.610044 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:45.649529 5196 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:45.743856 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 182583us
I20260929 14:12:45.744318 5196 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:45.745698 5316 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:45.648080 5347 raft_consensus.cc:1261] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260929 14:12:45.685349 5346 raft_consensus.cc:1261] T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260929 14:12:45.687638 8660 consensus_queue.cc:1035] T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:12:45.639426 8639 raft_consensus.cc:504] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:45.707330 8656 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:45.709036 5274 raft_consensus.cc:1261] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260929 14:12:45.646576 8554 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:45.669108 5271 raft_consensus.cc:2454] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
W20260929 14:12:45.652602 8668 log.cc:924] Time spent T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.067s user 0.000s sys 0.003s
I20260929 14:12:45.700646 5096 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:52750 (request call id 226) took 1008 ms. Trace:
I20260929 14:12:45.723135 5198 raft_consensus.cc:1261] T 9cd4876c229b4b768fe4180899274e70 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260929 14:12:45.724444 5199 raft_consensus.cc:1261] T 6239036de00f41c1b2ec5537a1ddc982 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260929 14:12:45.749889 5348 raft_consensus.cc:1261] T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260929 14:12:45.753401 5098 catalog_manager.cc:5495] T a3e77685e3424d0da5706ab4f0581529 P ac17d3b553b74c9da39e0ab307addacd reported cstate change: term changed from 7 to 8, leader changed from c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66) to ac17d3b553b74c9da39e0ab307addacd (127.3.242.65). New cstate: current_term: 8 leader_uuid: "ac17d3b553b74c9da39e0ab307addacd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } health_report { overall_health: UNKNOWN } } }
I20260929 14:12:45.782793 5096 rpcz_store.cc:276] 0929 14:12:44.692117 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:44.692385 (+ 268us) service_pool.cc:227] Handling call
0929 14:12:45.700566 (+1008181us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0929 14:12:44.880790 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0929 14:12:44.965185 (+ 84395us) write_op.cc:432] Acquiring schema lock in shared mode
0929 14:12:44.965242 (+ 57us) write_op.cc:435] Acquired schema lock
0929 14:12:44.965260 (+ 18us) tablet.cc:660] Decoding operations
0929 14:12:44.965637 (+ 377us) write_op.cc:620] Acquiring the partition lock for write op
0929 14:12:44.965711 (+ 74us) write_op.cc:641] Partition lock acquired for write op
0929 14:12:44.965746 (+ 35us) tablet.cc:683] Acquiring locks for 1 operations
0929 14:12:44.965955 (+ 209us) tablet.cc:699] Row locks acquired
0929 14:12:44.965968 (+ 13us) write_op.cc:260] PREPARE: finished
0929 14:12:44.966125 (+ 157us) write_op.cc:270] Start()
0929 14:12:44.966237 (+ 112us) write_op.cc:276] Timestamp: P: 1790691164966074 usec, L: 0
0929 14:12:44.966252 (+ 15us) op_driver.cc:331] REPLICATION: starting
0929 14:12:44.980876 (+ 14624us) log.cc:842] Serialized 538 byte log entry
0929 14:12:45.197137 (+216261us) op_driver.cc:444] REPLICATION: finished
0929 14:12:45.197511 (+ 374us) write_op.cc:301] APPLY: starting
0929 14:12:45.197589 (+ 78us) tablet.cc:1365] starting BulkCheckPresence
0929 14:12:45.197683 (+ 94us) tablet.cc:1368] finished BulkCheckPresence
0929 14:12:45.197779 (+ 96us) tablet.cc:1370] starting ApplyRowOperation cycle
0929 14:12:45.197980 (+ 201us) tablet.cc:1381] finished ApplyRowOperation cycle
0929 14:12:45.198123 (+ 143us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0929 14:12:45.198167 (+ 44us) write_op.cc:312] APPLY: finished
0929 14:12:45.198356 (+ 189us) log.cc:842] Serialized 23 byte log entry
0929 14:12:45.210168 (+ 11812us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:45.222499 (+ 12331us) write_op.cc:454] Released schema lock
0929 14:12:45.228265 (+ 5766us) write_op.cc:341] FINISH: Updating metrics
0929 14:12:45.250693 (+ 22428us) write_op.cc:489] Releasing partition, row and schema locks
0929 14:12:45.252465 (+ 1772us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2518,"reported_tablets":11,"child_traces":[["op",{"apply.queue_time_us":126,"apply.run_cpu_time_us":2050,"apply.run_wall_time_us":76286,"num_ops":1,"prepare.queue_time_us":474,"prepare.run_cpu_time_us":3194,"prepare.run_wall_time_us":110836,"replication_time_us":230778}]]}
I20260929 14:12:45.754550 8337 raft_consensus.cc:2726] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:45.782279 8545 raft_consensus.cc:2781] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260929 14:12:45.809201 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 183187us
I20260929 14:12:45.781539 8535 raft_consensus.cc:2726] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639) with higher term. Message: Invalid argument: T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 7. Current term is 8.
W20260929 14:12:45.812698 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 149225us
I20260929 14:12:45.816221 5347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:45.825245 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:45.730307 8663 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:45.809721 8545 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:45.793145 8499 consensus_queue.cc:1035] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260929 14:12:45.845340 5199 raft_consensus.cc:1261] T dbca949bc56f4fb19a19b93566383d8d P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260929 14:12:45.797551 8539 consensus_queue.cc:1035] T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260929 14:12:45.844398 8667 consensus_queue.cc:1035] T 6978e56e8f49441795cc7f6b6a8f4cfe P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.045s
I20260929 14:12:45.848984 8654 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:45.849363 5344 raft_consensus.cc:1261] T c15d9fe0c3724bfbad2156748acd468e P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Refusing update from remote peer 675b77a0a88a4bf08134281fc0cd949e: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260929 14:12:45.769603 8656 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:45.853966 8651 consensus_queue.cc:1035] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260929 14:12:45.838688 8496 raft_consensus.cc:465] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Not starting pre-election -- already a leader
I20260929 14:12:45.861495 8666 consensus_queue.cc:1035] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 14:12:45.785774 8674 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:45.863253 5194 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4"
dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
mode: GRACEFUL
new_leader_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
from {username='slave'} at 127.0.0.1:36470
I20260929 14:12:45.890775 5194 raft_consensus.cc:595] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Received request to transfer leadership to TS d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:45.870103 8454 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:45.871134 8639 leader_election.cc:288] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:45.871968 8640 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:45.754791 8672 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:45.890766 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 91098.2us
I20260929 14:12:45.847572 8545 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Advancing to term 8
W20260929 14:12:45.964617 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 71729.5us
I20260929 14:12:45.891901 5348 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 11.
I20260929 14:12:45.892076 5276 raft_consensus.cc:2379] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 11: Already voted for candidate c5ec116e39144881bdf1f2cd3e923cf5 in this term.
W20260929 14:12:45.963460 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 73946.9us
W20260929 14:12:45.973006 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 131016us
I20260929 14:12:45.973881 5269 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
I20260929 14:12:45.978729 5267 raft_consensus.cc:1261] T dbca949bc56f4fb19a19b93566383d8d P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260929 14:12:45.980104 5164 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:45.981094 8433 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Leader election won for term 11
I20260929 14:12:45.981599 8433 raft_consensus.cc:686] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 11 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:45.981740 8487 consensus_queue.cc:1035] T c15d9fe0c3724bfbad2156748acd468e P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.009s
I20260929 14:12:45.982695 8433 consensus_queue.cc:237] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:46.000229 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:46.012923 4041 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260929 14:12:46.020402 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:45.951488 8654 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:46.066107 4041 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260929 14:12:46.103986 5350 raft_consensus.cc:2454] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:46.167019 4041 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260929 14:12:46.057772 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:46.168061 5199 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 10
I20260929 14:12:46.054581 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:46.112870 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:46.176420 5194 raft_consensus.cc:2379] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 8: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:46.184398 8537 consensus_queue.cc:1035] T dbca949bc56f4fb19a19b93566383d8d P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260929 14:12:46.045068 8647 consensus_meta.cc:287] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.615s user 0.007s sys 0.003s
I20260929 14:12:46.192005 5199 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 10.
I20260929 14:12:46.066268 8651 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:46.207770 5200 raft_consensus.cc:2379] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 5: Already voted for candidate ac17d3b553b74c9da39e0ab307addacd in this term.
I20260929 14:12:46.199165 5239 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:46.020823 8454 consensus_queue.cc:1035] T 6239036de00f41c1b2ec5537a1ddc982 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.185s
I20260929 14:12:46.004449 8535 consensus_queue.cc:1035] T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.199s
I20260929 14:12:46.644717 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:46.621289 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:46.648712 5199 raft_consensus.cc:2454] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 3.
I20260929 14:12:46.198236 8647 raft_consensus.cc:504] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:46.299729 8625 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:46.217439 8663 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:46.276875 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 64888.3us
I20260929 14:12:46.405628 8654 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:46.303036 8651 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:46.558046 5351 consensus_meta.cc:287] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.514s user 0.008s sys 0.003s
I20260929 14:12:46.659246 5351 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:46.659729 5272 raft_consensus.cc:2354] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd for earlier term 5. Current term is 6.
I20260929 14:12:46.394861 8675 raft_consensus.cc:2781] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:46.551855 8674 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:46.628432 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 328330us
I20260929 14:12:46.673848 8647 leader_election.cc:288] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:46.517917 8656 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:46.560405 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 291360us
I20260929 14:12:46.675457 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:46.676119 5904 raft_consensus.cc:2454] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 4.
I20260929 14:12:46.677785 8625 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:46.283581 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 108002us
I20260929 14:12:46.520035 8672 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:46.681691 5271 raft_consensus.cc:2454] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 4.
I20260929 14:12:46.687794 5351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:46.688480 5351 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 11.
I20260929 14:12:46.702688 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:46.708227 8625 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:46.711709 8671 raft_consensus.cc:979] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd: : Instructing follower d886fb1eb0474d2d9f7410c4013c45e4 to start an election
I20260929 14:12:46.696762 8675 raft_consensus.cc:686] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:46.717111 5276 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 2228) took 1052 ms. Trace:
I20260929 14:12:46.717697 8675 consensus_queue.cc:237] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:46.717549 5276 rpcz_store.cc:276] 0929 14:12:45.664753 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:45.665189 (+ 436us) service_pool.cc:227] Handling call
0929 14:12:46.716878 (+1051689us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b5400392454. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:46.717023 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":334402432}
I20260929 14:12:46.687209 8698 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:46.722530 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:46.727105 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:46.723657 5167 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:12:46.730121 5904 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Advancing to term 6
W20260929 14:12:46.725909 5195 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s user 0.003s sys 0.014s
W20260929 14:12:46.731210 5195 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s user 0.003s sys 0.015s
W20260929 14:12:46.721915 8592 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.006s sys 0.010s
I20260929 14:12:46.739850 5239 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:46.747601 5239 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:46.735304 8641 raft_consensus.cc:1067] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Signalling peer d886fb1eb0474d2d9f7410c4013c45e4 to start an election
I20260929 14:12:46.752697 8663 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260929 14:12:46.741441 8592 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.555s user 0.007s sys 0.010s
I20260929 14:12:46.749015 8535 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:46.774180 8535 raft_consensus.cc:686] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:46.775416 8535 consensus_queue.cc:237] T 2818f47d8c6b41c1ab134b96096cacdc P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:46.781294 8663 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:46.781646 5904 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P 675b77a0a88a4bf08134281fc0cd949e [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
I20260929 14:12:46.781745 8663 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Advancing to term 12
I20260929 14:12:46.782706 5242 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:46.783533 8648 raft_consensus.cc:2781] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader election won for term 6
I20260929 14:12:46.783964 8648 raft_consensus.cc:686] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:46.784818 8648 consensus_queue.cc:237] T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:46.744748 8671 raft_consensus.cc:2781] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 14:12:46.789278 8671 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:46.743633 8698 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:46.817480 8650 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:46.818360 8671 raft_consensus.cc:3037] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 3 FOLLOWER]: Advancing to term 4
I20260929 14:12:46.818045 8650 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:46.818032 8433 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:46.820139 8433 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:46.820613 8650 leader_election.cc:288] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:46.823989 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:46.824095 8433 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:46.824668 5199 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 7.
I20260929 14:12:46.826100 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:46.826686 5194 raft_consensus.cc:3037] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Advancing to term 6
I20260929 14:12:46.830480 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3673029b370458db49bab90bbd0bd82" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:46.831030 5193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:46.835966 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:46.833411 5242 leader_election.cc:302] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters:
I20260929 14:12:46.831066 5346 raft_consensus.cc:3037] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:46.836585 5343 raft_consensus.cc:2454] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 5.
I20260929 14:12:46.841697 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:46.843986 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:46.842437 5343 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 8.
W20260929 14:12:46.816995 8545 consensus_meta.cc:287] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.817s user 0.007s sys 0.003s
I20260929 14:12:46.853708 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:46.863773 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:46.864795 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:46.910660 5345 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4"
dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
from {username='slave'} at 127.0.0.1:47252
I20260929 14:12:46.911294 5345 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 14:12:46.911655 5345 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:46.838059 8698 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:46.914973 5276 raft_consensus.cc:1261] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260929 14:12:46.924304 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:46.924364 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 89371.3us
I20260929 14:12:46.926640 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:46.927559 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:46.929263 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:46.859777 8545 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:46.919740 8651 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:46.929806 8660 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:47.001776 5194 raft_consensus.cc:2454] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 6.
W20260929 14:12:47.002498 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 49700.9us
I20260929 14:12:46.976531 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f84af62485a4405f887bb8675221641e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:47.004815 5873 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd; no voters:
I20260929 14:12:47.005841 5347 raft_consensus.cc:2379] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac17d3b553b74c9da39e0ab307addacd in current term 8: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
W20260929 14:12:46.981905 8704 log.cc:924] Time spent T 9cd4876c229b4b768fe4180899274e70 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20260929 14:12:47.014722 8660 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.031409 8660 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:47.034714 5164 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:47.036101 5164 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters: c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:47.071815 5343 raft_consensus.cc:1261] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Refusing update from remote peer ac17d3b553b74c9da39e0ab307addacd: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260929 14:12:47.077881 8494 raft_consensus.cc:2781] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260929 14:12:47.078732 8494 raft_consensus.cc:482] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:47.079176 8494 raft_consensus.cc:3037] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Advancing to term 5
W20260929 14:12:47.084805 5870 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 296554us
I20260929 14:12:47.003387 8698 raft_consensus.cc:482] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:47.037274 8686 raft_consensus.cc:2781] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader election won for term 10
W20260929 14:12:47.088598 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 45018.3us
I20260929 14:12:47.103194 5346 raft_consensus.cc:2454] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 10.
I20260929 14:12:47.106494 5346 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47258 (request call id 2093) took 1107 ms. Trace:
I20260929 14:12:47.073331 8645 raft_consensus.cc:2781] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260929 14:12:47.070254 8525 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:47.107054 5346 rpcz_store.cc:276] 0929 14:12:45.998611 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:45.998722 (+ 111us) service_pool.cc:227] Handling call
0929 14:12:46.830339 (+831617us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b5800183998. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda1989f11 00007fbda1b0d851 00007fbda1b0d702 00007fbdaa9fdb09 00007fbda9939730 00007fbda99ff79b 00007fbda99ff821 00007fbda99af1b0 00007fbda99aea94 00007fbda3096c04 00007fbda3096b92
0929 14:12:47.105446 (+275107us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5673728}
W20260929 14:12:47.114796 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 324453us
W20260929 14:12:47.115899 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 197018us
I20260929 14:12:47.122745 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:47.103868 8661 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.916s user 0.026s sys 0.027s
I20260929 14:12:47.086783 8496 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:47.146237 8707 log.cc:924] Time spent T 6978e56e8f49441795cc7f6b6a8f4cfe P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.219s user 0.000s sys 0.001s
I20260929 14:12:47.116876 8645 raft_consensus.cc:482] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:47.146977 8645 raft_consensus.cc:3037] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:47.116724 8686 raft_consensus.cc:686] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
W20260929 14:12:47.150444 8711 log.cc:924] Time spent T 2282d5a97d5a49b9a71bc53d08a7cb65 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.344s user 0.000s sys 0.001s
I20260929 14:12:47.116680 8698 raft_consensus.cc:504] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:47.117925 8525 raft_consensus.cc:482] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 14:12:47.152117 8525 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Advancing to term 9
W20260929 14:12:47.138685 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 133970us
W20260929 14:12:47.188151 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 270941us
W20260929 14:12:47.189183 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 59324.4us
W20260929 14:12:47.190037 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 48294.5us
I20260929 14:12:47.190387 5345 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:47.140944 8661 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.966s user 0.026s sys 0.027s
W20260929 14:12:47.225265 5871 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 213391us
I20260929 14:12:47.226135 5345 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:47.267380 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 58988us
I20260929 14:12:47.189342 8496 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:47.266654 5873 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 74306.9us
I20260929 14:12:47.288084 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: true dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:47.292758 5270 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:47.268405 8545 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:47.310675 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 79525.2us
I20260929 14:12:47.331647 5164 leader_election.cc:302] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:47.331908 5242 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:47.315280 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:47.340008 5192 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:47.346794 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e4f58231414bb5a9b16e093027d0c4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: true dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:47.350508 5197 raft_consensus.cc:3032] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Stepping down as leader of term 9
I20260929 14:12:47.350904 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:47.351457 5197 raft_consensus.cc:729] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:47.331408 8545 raft_consensus.cc:482] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:47.332741 8661 raft_consensus.cc:482] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
W20260929 14:12:47.350447 8719 log.cc:924] Time spent T 9cd4876c229b4b768fe4180899274e70 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.130s user 0.003s sys 0.000s
I20260929 14:12:47.369206 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01b3432723044cbab3d1437e32a2efda" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
W20260929 14:12:47.369990 8663 consensus_meta.cc:287] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5: Time spent flushing consensus metadata: real 0.588s user 0.013s sys 0.000s
I20260929 14:12:47.370370 8663 raft_consensus.cc:504] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.372397 8663 leader_election.cc:288] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 12 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:47.373636 5275 raft_consensus.cc:1226] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 12 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 11. Current term is 12. Ops: []
I20260929 14:12:47.348078 8706 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:47.376511 8706 raft_consensus.cc:504] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.377327 5349 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:38536 (request call id 595) took 1192 ms. Trace:
I20260929 14:12:47.378227 8730 raft_consensus.cc:2781] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260929 14:12:47.401779 5192 raft_consensus.cc:2454] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:47.406108 8706 leader_election.cc:288] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:47.285081 8686 consensus_queue.cc:237] T a3673029b370458db49bab90bbd0bd82 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.416097 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:47.444664 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:12:47.445430 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 113911us
I20260929 14:12:47.352240 5197 consensus_queue.cc:260] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.447567 5343 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Advancing to term 12
I20260929 14:12:47.449295 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:47.454579 5197 raft_consensus.cc:3037] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:47.446178 5316 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:47.377743 5349 rpcz_store.cc:276] 0929 14:12:46.099299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:46.099450 (+ 151us) service_pool.cc:227] Handling call
0929 14:12:46.174598 (+ 75148us) raft_consensus.cc:1457] Updating replica for 1 ops
0929 14:12:46.204431 (+ 29833us) raft_consensus.cc:1520] Early marking committed up to index 0
0929 14:12:46.204463 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0929 14:12:46.277885 (+ 73422us) log.cc:842] Serialized 26 byte log entry
0929 14:12:46.730514 (+452629us) raft_consensus.cc:1631] Marking committed up to 6
0929 14:12:46.730562 (+ 48us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:46.730640 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0929 14:12:47.291755 (+561115us) raft_consensus.cc:1671] finished
0929 14:12:47.291773 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:47.291898 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":294812,"raft.queue_time_us":103816,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":36249,"spinlock_wait_cycles":2304,"thread_start_us":369179,"threads_started":2,"wal-append.queue_time_us":437397}
I20260929 14:12:47.457063 5350 raft_consensus.cc:1261] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260929 14:12:47.411996 8650 raft_consensus.cc:2726] T f84af62485a4405f887bb8675221641e P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:47.477649 5343 raft_consensus.cc:2454] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 12.
I20260929 14:12:47.378727 8730 raft_consensus.cc:482] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Starting leader election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:47.398032 8545 raft_consensus.cc:504] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:47.539752 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 74616.3us
I20260929 14:12:47.445078 8660 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:12:47.472191 8671 consensus_meta.cc:287] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.653s user 0.015s sys 0.001s
I20260929 14:12:47.413776 8661 raft_consensus.cc:504] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.520480 8698 leader_election.cc:288] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:47.523638 8748 log.cc:924] Time spent T 5e18ad4fc5524f4687d35e8b2dd03193 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.145s user 0.001s sys 0.000s
I20260929 14:12:47.513357 8688 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:47.516115 8739 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:47.518124 8743 consensus_queue.cc:1035] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260929 14:12:47.521364 8720 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:47.522110 8726 raft_consensus.cc:482] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:47.532620 5197 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 10.
I20260929 14:12:47.538189 5270 raft_consensus.cc:2454] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 10.
I20260929 14:12:47.539542 8730 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 6 FOLLOWER]: Advancing to term 7
I20260929 14:12:47.588450 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:47.540772 8525 raft_consensus.cc:504] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:47.598172 8735 log.cc:924] Time spent T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
I20260929 14:12:47.599310 8525 leader_election.cc:288] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:47.602759 8545 leader_election.cc:288] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:47.602461 8720 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:47.536187 8494 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:47.604576 8720 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:47.605211 8494 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:47.605049 8726 raft_consensus.cc:504] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:47.607173 8726 leader_election.cc:288] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:47.608438 8726 raft_consensus.cc:482] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:47.555593 8660 raft_consensus.cc:504] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:47.557469 8496 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:47.552006 8671 raft_consensus.cc:504] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:47.576795 8739 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.612150 5906 raft_consensus.cc:2379] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 5: Already voted for candidate 675b77a0a88a4bf08134281fc0cd949e in this term.
I20260929 14:12:47.618597 5193 raft_consensus.cc:2454] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 4.
W20260929 14:12:47.622646 5272 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50770 (request call id 2227) took 1960 ms (client timeout 1944 ms). Trace:
W20260929 14:12:47.640672 5272 rpcz_store.cc:269] 0929 14:12:45.662149 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:45.662759 (+ 610us) service_pool.cc:227] Handling call
0929 14:12:47.622433 (+1959674us) spinlock_profiling.cc:243] Waited 915 ms on lock 0x7b54003b0e54. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:47.622558 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2105009536}
I20260929 14:12:47.643055 5345 raft_consensus.cc:1226] T 4b277670ba5344b493c046b2a0864597 P d886fb1eb0474d2d9f7410c4013c45e4 [term 12 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260929 14:12:47.625190 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
W20260929 14:12:47.717849 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 82225.6us
W20260929 14:12:47.644572 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 169156us
W20260929 14:12:47.717897 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 271772us
I20260929 14:12:47.610918 8660 leader_election.cc:288] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
W20260929 14:12:47.720945 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 268370us
I20260929 14:12:47.724676 5344 raft_consensus.cc:3037] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:47.614288 8752 raft_consensus.cc:482] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:47.728714 8752 raft_consensus.cc:504] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:47.731499 8752 leader_election.cc:288] T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:47.617051 8661 leader_election.cc:288] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:47.735572 5194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:47.609396 8756 raft_consensus.cc:2781] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader election won for term 8
I20260929 14:12:47.734094 5314 leader_election.cc:302] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:47.740849 5197 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:47.742831 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e"
I20260929 14:12:47.619287 8671 leader_election.cc:288] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:47.745826 5199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:47.573976 8688 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.749279 8663 raft_consensus.cc:482] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:47.749898 8663 raft_consensus.cc:504] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.622066 8755 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:47.614171 8750 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
W20260929 14:12:47.724641 5165 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.3.242.66:43473 timed out after 1.944s (SENT) [suppressed 17 similar messages]
I20260929 14:12:47.627101 8739 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:47.753854 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 196600us
I20260929 14:12:47.755247 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19257f7a5d364214b38d26c5f9b6d190" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:47.756291 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:47.755066 5316 leader_election.cc:302] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:47.733922 8660 raft_consensus.cc:482] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:47.609167 8726 raft_consensus.cc:504] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:47.608470 8645 raft_consensus.cc:504] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.813989 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:47.750638 8688 leader_election.cc:288] T 465f57e1a1654ed1b40ce88c27fa52ac P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:47.811729 8663 leader_election.cc:288] T 4fc85ccb35394b2d800cea4f13552756 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:47.838629 8768 raft_consensus.cc:482] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:47.839046 8669 consensus_queue.cc:1035] T f7507b3dbd59424f8887991236a4a49c P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260929 14:12:47.759615 8671 consensus_queue.cc:1046] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.066s
I20260929 14:12:47.849421 8720 raft_consensus.cc:2781] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader election won for term 10
I20260929 14:12:47.854130 8766 raft_consensus.cc:482] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:47.854773 8720 raft_consensus.cc:686] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:47.855058 8766 raft_consensus.cc:504] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.855648 8720 consensus_queue.cc:237] T e9e4f58231414bb5a9b16e093027d0c4 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.811487 8660 raft_consensus.cc:504] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:47.760222 8755 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:47.899562 5167 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 151955us
W20260929 14:12:47.901031 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 176405us
I20260929 14:12:47.762444 8750 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.906709 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:47.907341 5345 raft_consensus.cc:3037] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Advancing to term 6
I20260929 14:12:47.909010 5165 leader_election.cc:302] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:47.911731 5344 raft_consensus.cc:2454] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 9.
I20260929 14:12:47.878521 8753 raft_consensus.cc:482] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:47.899212 8743 raft_consensus.cc:3032] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 11 LEADER]: Stepping down as leader of term 11
I20260929 14:12:47.919523 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:47.919981 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:47.920094 8751 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:47.921479 8763 raft_consensus.cc:2726] T 01b3432723044cbab3d1437e32a2efda P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260929 14:12:47.908290 8671 raft_consensus.cc:482] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:47.914014 8754 raft_consensus.cc:2726] T 44e89147efa646d3b112a40f53a086a3 P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:47.853205 8768 raft_consensus.cc:504] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.916355 5164 leader_election.cc:302] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:47.935963 5345 raft_consensus.cc:2454] T f7507b3dbd59424f8887991236a4a49c P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
W20260929 14:12:47.936802 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 181835us
I20260929 14:12:47.902311 8645 leader_election.cc:288] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:47.939715 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 205702us
I20260929 14:12:47.941155 8726 leader_election.cc:288] T 94e36a01dad24c1193b628e9200d3c2d P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:47.941711 8671 raft_consensus.cc:504] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:47.954425 5272 raft_consensus.cc:1226] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Rejecting Update request from peer ac17d3b553b74c9da39e0ab307addacd for earlier term 9. Current term is 10. Ops: []
I20260929 14:12:47.912861 8732 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:47.912848 8749 raft_consensus.cc:482] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:47.943923 8750 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:47.929929 8743 raft_consensus.cc:729] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 11 LEADER]: Becoming Follower/Learner. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:47.951444 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c3538be0f624dd79d0e667313b22d02" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:47.941826 8751 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:47.961329 5270 raft_consensus.cc:2454] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 9.
I20260929 14:12:47.957952 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2818f47d8c6b41c1ab134b96096cacdc" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:47.963517 8766 leader_election.cc:288] T f2610284ac4749c5bb443c221df8ff92 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:47.964388 8660 leader_election.cc:288] T 44e89147efa646d3b112a40f53a086a3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:47.967897 8685 consensus_queue.cc:1035] T 0c3538be0f624dd79d0e667313b22d02 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260929 14:12:47.967954 8730 raft_consensus.cc:504] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.974176 8755 leader_election.cc:288] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:47.975838 5275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:47.977082 5267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05a81ea24145409b82fe949d849ab73d" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:47.977312 5276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:47.977689 5267 raft_consensus.cc:2454] T 05a81ea24145409b82fe949d849ab73d P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 7.
I20260929 14:12:47.977938 5276 raft_consensus.cc:3037] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Advancing to term 5
I20260929 14:12:47.979121 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55a9c8ae48e44c59fee506e539b3f90" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:47.979215 8749 raft_consensus.cc:504] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.980810 5870 leader_election.cc:302] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:47.981357 8749 leader_election.cc:288] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:47.932111 8753 raft_consensus.cc:504] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.982000 8759 raft_consensus.cc:2781] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader election won for term 9
I20260929 14:12:47.982568 8759 raft_consensus.cc:686] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:47.961773 5316 leader_election.cc:302] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:47.983804 8753 leader_election.cc:288] T 879dc5894e734d098eb2d6064e10712a P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:47.951555 8756 raft_consensus.cc:686] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 LEADER]: Becoming Leader. State: Replica: d886fb1eb0474d2d9f7410c4013c45e4, State: Running, Role: LEADER
I20260929 14:12:47.984833 8768 leader_election.cc:288] T dfe03c3eac5d42c280945fcb723aabc3 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:47.985464 8756 consensus_queue.cc:237] T 3565847d030f4cc49b113969f438c801 P d886fb1eb0474d2d9f7410c4013c45e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.986940 8743 consensus_queue.cc:260] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.987943 8730 leader_election.cc:288] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 7 election: Requested vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:47.988194 8743 raft_consensus.cc:3037] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 11 FOLLOWER]: Advancing to term 12
I20260929 14:12:47.960855 8732 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:47.990813 8671 leader_election.cc:288] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:47.982021 8694 raft_consensus.cc:2726] T e55a9c8ae48e44c59fee506e539b3f90 P 675b77a0a88a4bf08134281fc0cd949e [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260929 14:12:47.991997 8732 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:47.954120 8525 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:47.993475 8525 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:47.995975 8525 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:47.998023 8751 leader_election.cc:288] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:47.983454 8759 consensus_queue.cc:237] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:48.007927 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:48.018558 8539 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 1.194s user 0.007s sys 0.018s
I20260929 14:12:47.954887 8688 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:48.026183 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88e8219d06734c04acdce83c7b9c1c53" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:48.026317 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08b2fdc7f66741049e11f24a1866c63f" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:48.029276 8747 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:48.024417 8539 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 1.200s user 0.007s sys 0.018s
I20260929 14:12:48.029907 8747 raft_consensus.cc:504] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:48.031905 5906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:48.032963 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2282d5a97d5a49b9a71bc53d08a7cb65" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:48.033632 5870 leader_election.cc:302] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:48.034739 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:48.035317 5316 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 73615.3us
I20260929 14:12:48.038790 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:48.037941 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:48.042053 5276 raft_consensus.cc:2454] T 08b2fdc7f66741049e11f24a1866c63f P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 5.
I20260929 14:12:48.042544 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "599240a72b454fca841864f021dbbc64" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:48.043493 5871 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd
W20260929 14:12:48.043916 5164 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 127636us
I20260929 14:12:48.044505 8528 raft_consensus.cc:2781] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 5 FOLLOWER]: Leader election won for term 5
I20260929 14:12:48.045007 8528 raft_consensus.cc:686] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [term 5 LEADER]: Becoming Leader. State: Replica: 675b77a0a88a4bf08134281fc0cd949e, State: Running, Role: LEADER
I20260929 14:12:48.046016 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:48.054167 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:48.056506 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c15d9fe0c3724bfbad2156748acd468e" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:48.057480 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:48.037185 8751 raft_consensus.cc:2726] T 88e8219d06734c04acdce83c7b9c1c53 P 675b77a0a88a4bf08134281fc0cd949e [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:48.063580 5164 leader_election.cc:302] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:48.065659 5272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f7507b3dbd59424f8887991236a4a49c" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5"
I20260929 14:12:48.047145 8749 raft_consensus.cc:2726] T 19257f7a5d364214b38d26c5f9b6d190 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:48.070606 5164 leader_election.cc:302] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:48.048987 8688 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:48.077605 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e285df817143ec9947058346e02b90" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:48.079706 8688 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:48.046031 8528 consensus_queue.cc:237] T 08b2fdc7f66741049e11f24a1866c63f P 675b77a0a88a4bf08134281fc0cd949e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:48.093776 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22fb3fae8e7947989abe31ce58000d58" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:48.097080 5269 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:50776 (request call id 2278) took 2429 ms (client timeout 2337 ms). Trace:
I20260929 14:12:48.071964 8496 raft_consensus.cc:2781] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader election won for term 6
W20260929 14:12:48.097683 5269 rpcz_store.cc:269] 0929 14:12:45.660583 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:45.660768 (+ 185us) service_pool.cc:227] Handling call
0929 14:12:48.090008 (+2429240us) spinlock_profiling.cc:243] Waited 235 ms on lock 0x7b54003368d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:48.090145 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":539625984}
I20260929 14:12:48.102721 5269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:48.103649 5271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:48.103319 5314 leader_election.cc:302] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:48.106827 5314 leader_election.cc:302] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:48.107934 5314 leader_election.cc:302] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:48.104408 5271 raft_consensus.cc:2454] T 632bdccb08444a4d89ecc1cfdbab98b2 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 8.
I20260929 14:12:48.109104 5314 leader_election.cc:302] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:48.114220 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:48.121083 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:48.084465 8658 raft_consensus.cc:482] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:48.126550 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b277670ba5344b493c046b2a0864597" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:48.130172 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4300e83384e9458bb280a4b1e3131932" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:48.131750 5345 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:48.136412 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:48.137449 5164 leader_election.cc:302] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:48.126117 8720 raft_consensus.cc:2726] T 2282d5a97d5a49b9a71bc53d08a7cb65 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:48.146797 8670 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:48.149665 8670 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:48.150807 5314 leader_election.cc:302] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d886fb1eb0474d2d9f7410c4013c45e4; no voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:48.151674 8670 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:48.152292 5314 leader_election.cc:302] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters: 675b77a0a88a4bf08134281fc0cd949e
I20260929 14:12:48.156383 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d653defbcc7e47ca89b35b1383e4e2c8" candidate_uuid: "ac17d3b553b74c9da39e0ab307addacd" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:48.124099 8651 raft_consensus.cc:482] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:48.156862 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:48.126006 8732 raft_consensus.cc:2726] T 599240a72b454fca841864f021dbbc64 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260929 14:12:48.161077 8700 raft_consensus.cc:2726] T dfe03c3eac5d42c280945fcb723aabc3 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:48.167781 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4dd0a13d8aa4e6bbd3633c047984f67" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:48.170521 5348 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:47252 (request call id 2527) took 1233 ms. Trace:
I20260929 14:12:48.174331 5198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:48.169435 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:48.163594 8670 raft_consensus.cc:2726] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:48.199010 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:48.166638 8651 raft_consensus.cc:504] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:48.195842 8768 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:48.201349 8778 raft_consensus.cc:2726] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260929 14:12:48.195380 5165 leader_election.cc:302] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:48.171036 5348 rpcz_store.cc:276] 0929 14:12:46.936737 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:46.937054 (+ 317us) service_pool.cc:227] Handling call
0929 14:12:46.971521 (+ 34467us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:46.971731 (+ 210us) raft_consensus.cc:1520] Early marking committed up to index 5
0929 14:12:47.087419 (+115688us) log.cc:842] Serialized 14 byte log entry
0929 14:12:48.169952 (+1082533us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b18003b7e20. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbdaa9e48b1 00007fbdaa9f1661 00007fbda46fc82c 00007fbda1b279dc 00007fbdaa95ea39 00007fbda1b448bb 00007fbda1b42cbf 00007fbda1b42155 00007fbda4a653d0 00007fbda4a77576 00007fbda4a880d7
0929 14:12:48.170076 (+ 124us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:48.170141 (+ 65us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:48.170218 (+ 77us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:48.170311 (+ 93us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:48.170452 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":59366,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"raft.queue_time_us":86366,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":11776,"spinlock_wait_cycles":4231040,"thread_start_us":339438,"threads_started":3,"wal-append.queue_time_us":329428}
I20260929 14:12:48.201040 8768 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:48.204656 5165 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd, c5ec116e39144881bdf1f2cd3e923cf5; no voters: d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:48.206440 8651 leader_election.cc:288] T 7e9a6f1c02ae4024813fda5af5c665da P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:48.207679 8627 raft_consensus.cc:2781] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260929 14:12:48.208225 8627 raft_consensus.cc:482] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:48.206233 5242 leader_election.cc:302] T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: ac17d3b553b74c9da39e0ab307addacd, d886fb1eb0474d2d9f7410c4013c45e4
I20260929 14:12:48.214335 5165 leader_election.cc:302] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
I20260929 14:12:48.154444 8658 raft_consensus.cc:504] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:48.211864 8627 raft_consensus.cc:3037] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Advancing to term 8
I20260929 14:12:48.218529 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:48.211553 8753 raft_consensus.cc:2781] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260929 14:12:48.157608 8739 raft_consensus.cc:2726] T 22fb3fae8e7947989abe31ce58000d58 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260929 14:12:48.269734 5165 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 45252.3us
I20260929 14:12:48.359220 5165 leader_election.cc:302] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ac17d3b553b74c9da39e0ab307addacd; no voters: 675b77a0a88a4bf08134281fc0cd949e, c5ec116e39144881bdf1f2cd3e923cf5
W20260929 14:12:48.324292 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 97862.1us
I20260929 14:12:48.166802 8720 raft_consensus.cc:2781] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260929 14:12:48.368993 8720 raft_consensus.cc:482] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:48.369558 8720 raft_consensus.cc:3037] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Advancing to term 9
I20260929 14:12:48.372287 8768 leader_election.cc:288] T a3e77685e3424d0da5706ab4f0581529 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:48.379323 5350 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac17d3b553b74c9da39e0ab307addacd in term 6.
I20260929 14:12:48.217468 8645 raft_consensus.cc:2726] T 08b2fdc7f66741049e11f24a1866c63f P ac17d3b553b74c9da39e0ab307addacd [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260929 14:12:48.388789 8749 raft_consensus.cc:482] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:48.396312 8749 raft_consensus.cc:504] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:48.396970 8746 raft_consensus.cc:482] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:48.397522 8746 raft_consensus.cc:504] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:48.398311 8749 leader_election.cc:288] T 929c62a6106e42d08a13689e62dce3b1 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:48.399151 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:48.399307 8454 raft_consensus.cc:482] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:48.399511 8746 leader_election.cc:288] T 4040985909414481ad0d1491249e8e67 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:48.400373 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:48.400532 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44e89147efa646d3b112a40f53a086a3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:48.406458 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:48.243420 8753 raft_consensus.cc:482] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Starting leader election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:48.408483 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd"
I20260929 14:12:48.409300 5192 raft_consensus.cc:2354] T 3565847d030f4cc49b113969f438c801 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 7. Current term is 8.
I20260929 14:12:48.416754 8658 leader_election.cc:288] T 28e681a7ebc549789ec0a6f1a75c45d9 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:48.416986 5192 raft_consensus.cc:1261] T a3673029b370458db49bab90bbd0bd82 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260929 14:12:48.378424 8661 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:48.399853 8454 raft_consensus.cc:504] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:48.384531 8747 leader_election.cc:288] T b7924b2689dc46c98af8c4344a1e1e33 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:48.393805 8764 raft_consensus.cc:482] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
W20260929 14:12:48.485260 5239 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.668s user 0.015s sys 0.034s
W20260929 14:12:48.485707 5239 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.672s user 0.015s sys 0.034s
W20260929 14:12:48.486006 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 692606us
W20260929 14:12:48.545295 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.65:45147, user_credentials={real_user=slave}} blocked reactor thread for 147314us
I20260929 14:12:48.409919 8645 raft_consensus.cc:2726] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260929 14:12:48.410104 8753 raft_consensus.cc:3037] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd [term 9 FOLLOWER]: Advancing to term 10
I20260929 14:12:48.408447 8641 raft_consensus.cc:482] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:48.470245 8746 raft_consensus.cc:482] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:48.635583 5901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "675b77a0a88a4bf08134281fc0cd949e" is_pre_election: true
I20260929 14:12:48.658694 8648 raft_consensus.cc:482] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:48.574044 8756 raft_consensus.cc:482] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:48.662147 5242 leader_election.cc:302] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:48.471334 8661 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:48.556581 8764 raft_consensus.cc:504] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:48.725775 8778 raft_consensus.cc:482] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:48.635587 8641 raft_consensus.cc:504] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:48.769121 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 247693us
W20260929 14:12:48.722470 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 60429us
I20260929 14:12:48.786146 5242 leader_election.cc:302] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5; no voters: 675b77a0a88a4bf08134281fc0cd949e, ac17d3b553b74c9da39e0ab307addacd
I20260929 14:12:48.790058 5239 leader_election.cc:302] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5ec116e39144881bdf1f2cd3e923cf5, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
I20260929 14:12:48.792193 8750 raft_consensus.cc:2726] T d4dd0a13d8aa4e6bbd3633c047984f67 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260929 14:12:48.693925 8648 raft_consensus.cc:504] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:48.666700 8683 raft_consensus.cc:482] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 12 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:48.731848 8788 log.cc:924] Time spent T 6978e56e8f49441795cc7f6b6a8f4cfe P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.323s user 0.000s sys 0.000s
I20260929 14:12:48.797627 8683 raft_consensus.cc:504] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:48.732188 8778 raft_consensus.cc:504] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:48.800020 8778 leader_election.cc:288] T 6239036de00f41c1b2ec5537a1ddc982 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:48.801018 8787 log.cc:924] Time spent T 91e285df817143ec9947058346e02b90 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.588s user 0.000s sys 0.001s
I20260929 14:12:48.659263 8774 raft_consensus.cc:482] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:48.802771 8774 raft_consensus.cc:504] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:48.804854 8648 leader_election.cc:288] T e5acb0315ba44b5a9d145119868e7168 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:48.804854 8774 leader_election.cc:288] T 5e18ad4fc5524f4687d35e8b2dd03193 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:48.799650 8683 leader_election.cc:288] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:48.806288 8641 leader_election.cc:288] T 01b3432723044cbab3d1437e32a2efda P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:48.734652 8693 raft_consensus.cc:2726] T 4300e83384e9458bb280a4b1e3131932 P c5ec116e39144881bdf1f2cd3e923cf5 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260929 14:12:48.815029 8743 consensus_meta.cc:287] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 0.826s user 0.012s sys 0.000s
I20260929 14:12:48.816432 5201 raft_consensus.cc:2396] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 for term 12 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260929 14:12:48.819671 8661 leader_election.cc:288] T 625b1b3f659041f8b54de3499628ebc4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
I20260929 14:12:48.667052 8746 raft_consensus.cc:504] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:48.727455 8756 raft_consensus.cc:504] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:48.732826 8764 leader_election.cc:288] T 62fed7cb04b44fca9312372e511f1c3e P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
W20260929 14:12:48.890450 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 738192us
I20260929 14:12:48.893746 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:48.895115 8454 leader_election.cc:288] T a39f12a08e914559b1a12cbac3e3cec4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:48.895937 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:48.898798 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:48.901844 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "625b1b3f659041f8b54de3499628ebc4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:48.908439 8746 leader_election.cc:288] T 58eff9a0aa514df1999bdd22fb513354 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473)
W20260929 14:12:48.890481 5242 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.68:35497, user_credentials={real_user=slave}} blocked reactor thread for 104446us
W20260929 14:12:48.951594 5239 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.3.242.67:42639, user_credentials={real_user=slave}} blocked reactor thread for 161629us
I20260929 14:12:48.983299 8780 raft_consensus.cc:2781] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 12 FOLLOWER]: Leader election won for term 12
I20260929 14:12:48.983964 8780 raft_consensus.cc:686] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [term 12 LEADER]: Becoming Leader. State: Replica: c5ec116e39144881bdf1f2cd3e923cf5, State: Running, Role: LEADER
I20260929 14:12:48.984856 8780 consensus_queue.cc:237] T 4b277670ba5344b493c046b2a0864597 P c5ec116e39144881bdf1f2cd3e923cf5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:48.986977 5200 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52188 (request call id 1902) took 2898 ms (client timeout 1955 ms). Trace:
W20260929 14:12:48.987407 5200 rpcz_store.cc:269] 0929 14:12:46.088541 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:46.088684 (+ 143us) service_pool.cc:227] Handling call
0929 14:12:48.986742 (+2898058us) spinlock_profiling.cc:243] Waited 37.2 ms on lock 0x7b5400408654. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:48.986887 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":85665664}
I20260929 14:12:48.988662 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:48.970685 8661 raft_consensus.cc:482] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
W20260929 14:12:48.989782 5193 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52188 (request call id 1915) took 2159 ms (client timeout 1994 ms). Trace:
I20260929 14:12:48.989957 8661 raft_consensus.cc:504] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:48.990172 5193 rpcz_store.cc:269] 0929 14:12:46.830560 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:46.830700 (+ 140us) service_pool.cc:227] Handling call
0929 14:12:48.989627 (+2158927us) spinlock_profiling.cc:243] Waited 787 ms on lock 0x7b540031e054. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:48.989723 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1809379584}
I20260929 14:12:48.991946 8661 leader_election.cc:288] T 520249ed5be343dcb30c86d3396491a4 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:48.851820 8627 raft_consensus.cc:504] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:49.022496 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6978e56e8f49441795cc7f6b6a8f4cfe" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:49.047286 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "465f57e1a1654ed1b40ce88c27fa52ac" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:49.047317 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:49.048362 5200 raft_consensus.cc:1261] T 91e285df817143ec9947058346e02b90 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Refusing update from remote peer c5ec116e39144881bdf1f2cd3e923cf5: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260929 14:12:49.080796 5350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fc85ccb35394b2d800cea4f13552756" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:49.112903 8720 consensus_meta.cc:287] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4: Time spent flushing consensus metadata: real 0.743s user 0.007s sys 0.004s
W20260929 14:12:49.224746 8496 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.080s user 0.037s sys 0.082s
I20260929 14:12:49.140213 8720 raft_consensus.cc:504] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:49.628170 5273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "520249ed5be343dcb30c86d3396491a4" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
I20260929 14:12:49.265707 5269 raft_consensus.cc:3037] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Advancing to term 8
W20260929 14:12:49.632221 5314 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.3.242.66:43473, user_credentials={real_user=slave}} blocked reactor thread for 166986us
W20260929 14:12:49.458778 5276 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.332s user 0.018s sys 0.019s
I20260929 14:12:49.304036 5270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58eff9a0aa514df1999bdd22fb513354" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" is_pre_election: true
W20260929 14:12:49.682312 5276 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.556s user 0.022s sys 0.023s
I20260929 14:12:49.255957 8661 raft_consensus.cc:482] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:49.702661 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e36a01dad24c1193b628e9200d3c2d" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:49.227305 8528 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.143s user 0.037s sys 0.085s
I20260929 14:12:49.227074 8589 raft_consensus.cc:482] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:49.702678 5270 raft_consensus.cc:2454] T 58eff9a0aa514df1999bdd22fb513354 P c5ec116e39144881bdf1f2cd3e923cf5 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 9.
I20260929 14:12:49.703609 5344 raft_consensus.cc:2454] T 94e36a01dad24c1193b628e9200d3c2d P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 5.
I20260929 14:12:49.258167 8747 raft_consensus.cc:482] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
W20260929 14:12:49.761945 5166 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.865s user 0.030s sys 0.035s
I20260929 14:12:49.761773 8747 raft_consensus.cc:504] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:49.762408 5166 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.210s user 0.030s sys 0.035s
I20260929 14:12:49.762809 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfe03c3eac5d42c280945fcb723aabc3" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:49.764124 8747 leader_election.cc:288] T d653defbcc7e47ca89b35b1383e4e2c8 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:49.765492 5348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "879dc5894e734d098eb2d6064e10712a" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:49.766969 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3565847d030f4cc49b113969f438c801" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4"
I20260929 14:12:49.768574 5349 raft_consensus.cc:1226] T 2818f47d8c6b41c1ab134b96096cacdc P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9. Ops: []
W20260929 14:12:49.626047 5313 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.100s user 0.029s sys 0.044s
W20260929 14:12:49.264909 8759 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.157s user 0.027s sys 0.038s
I20260929 14:12:49.774472 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2610284ac4749c5bb443c221df8ff92" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:49.777678 5344 raft_consensus.cc:2454] T f2610284ac4749c5bb443c221df8ff92 P d886fb1eb0474d2d9f7410c4013c45e4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 11.
W20260929 14:12:49.731172 8528 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.647s user 0.037s sys 0.088s
I20260929 14:12:49.458508 8778 raft_consensus.cc:482] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:49.811061 5344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e9a6f1c02ae4024813fda5af5c665da" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:49.398528 8539 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.368s user 0.010s sys 0.038s
I20260929 14:12:49.779606 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "632bdccb08444a4d89ecc1cfdbab98b2" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:49.826623 5343 raft_consensus.cc:2379] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c5ec116e39144881bdf1f2cd3e923cf5 in current term 9: Already voted for candidate d886fb1eb0474d2d9f7410c4013c45e4 in this term.
I20260929 14:12:49.752177 8589 raft_consensus.cc:504] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:49.780515 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7924b2689dc46c98af8c4344a1e1e33" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:49.781854 8649 raft_consensus.cc:482] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
I20260929 14:12:49.794080 8702 raft_consensus.cc:482] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader c5ec116e39144881bdf1f2cd3e923cf5)
W20260929 14:12:49.774487 5313 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.619s user 0.035s sys 0.050s
I20260929 14:12:49.703644 8647 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:49.428035 8627 leader_election.cc:288] T 05a81ea24145409b82fe949d849ab73d P ac17d3b553b74c9da39e0ab307addacd [CANDIDATE]: Term 8 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:49.834158 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6239036de00f41c1b2ec5537a1ddc982" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:49.621695 8749 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:49.847457 5871 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.122s user 0.015s sys 0.048s
W20260929 14:12:49.777485 8759 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.669s user 0.027s sys 0.059s
I20260929 14:12:49.821480 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e77685e3424d0da5706ab4f0581529" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:49.822505 5167 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.302s user 0.034s sys 0.085s
W20260929 14:12:49.852540 5167 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.332s user 0.034s sys 0.086s
W20260929 14:12:49.853025 5871 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.135s user 0.015s sys 0.049s
I20260929 14:12:49.776746 8688 raft_consensus.cc:482] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:49.731266 8661 raft_consensus.cc:504] T 678eb0db533d4bea872808b6fb39b764 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:49.848621 5193 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.856s user 0.011s sys 0.020s
I20260929 14:12:49.850391 5343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a39f12a08e914559b1a12cbac3e3cec4" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:49.851449 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4040985909414481ad0d1491249e8e67" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:49.852605 5349 raft_consensus.cc:2454] T a3e77685e3424d0da5706ab4f0581529 P d886fb1eb0474d2d9f7410c4013c45e4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ec116e39144881bdf1f2cd3e923cf5 in term 8.
I20260929 14:12:49.836246 8649 raft_consensus.cc:504] T 625b1b3f659041f8b54de3499628ebc4 P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:49.839021 8799 raft_consensus.cc:482] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:49.860445 5198 raft_consensus.cc:1226] T 2818f47d8c6b41c1ab134b96096cacdc P ac17d3b553b74c9da39e0ab307addacd [term 9 LEADER]: Rejecting Update request from peer c5ec116e39144881bdf1f2cd3e923cf5 for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:49.622267 8746 raft_consensus.cc:482] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:49.899374 5193 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.907s user 0.012s sys 0.025s
W20260929 14:12:49.899256 5870 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.359s user 0.051s sys 0.123s
W20260929 14:12:49.903218 5870 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.363s user 0.051s sys 0.123s
W20260929 14:12:49.904836 5164 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.381s user 0.026s sys 0.033s
W20260929 14:12:49.705876 8496 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.565s user 0.046s sys 0.094s
W20260929 14:12:49.905287 5164 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.382s user 0.026s sys 0.033s
I20260929 14:12:49.821856 8778 raft_consensus.cc:504] T 88e8219d06734c04acdce83c7b9c1c53 P d886fb1eb0474d2d9f7410c4013c45e4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
W20260929 14:12:49.826368 8539 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.796s user 0.013s sys 0.052s
W20260929 14:12:49.881402 8580 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 3.041s user 0.041s sys 0.052s
W20260929 14:12:49.873839 8780 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.760s user 0.001s sys 0.021s
I20260929 14:12:49.915998 8337 raft_consensus.cc:482] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:49.916534 5192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e681a7ebc549789ec0a6f1a75c45d9" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:49.917140 8589 leader_election.cc:288] T e9e4f58231414bb5a9b16e093027d0c4 P c5ec116e39144881bdf1f2cd3e923cf5 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers ac17d3b553b74c9da39e0ab307addacd (127.3.242.65:45147), d886fb1eb0474d2d9f7410c4013c45e4 (127.3.242.67:42639)
I20260929 14:12:49.841261 8647 raft_consensus.cc:504] T f84af62485a4405f887bb8675221641e P c5ec116e39144881bdf1f2cd3e923cf5 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } attrs { replace: true } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:49.923424 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e18ad4fc5524f4687d35e8b2dd03193" candidate_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
W20260929 14:12:49.881930 5272 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.657s user 0.012s sys 0.031s
W20260929 14:12:49.928262 5197 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.869s user 0.003s sys 0.049s
W20260929 14:12:49.919564 8580 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 3.087s user 0.042s sys 0.052s
I20260929 14:12:49.881834 5200 raft_consensus.cc:2454] T 4040985909414481ad0d1491249e8e67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:49.905493 8496 raft_consensus.cc:686] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [term 6 LEADER]: Becoming Leader. State: Replica: ac17d3b553b74c9da39e0ab307addacd, State: Running, Role: LEADER
I20260929 14:12:49.844115 8749 raft_consensus.cc:504] T ae31abec82ff4f40b695ccb19f289dbc P d886fb1eb0474d2d9f7410c4013c45e4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:49.882185 8759 raft_consensus.cc:482] T d4dd0a13d8aa4e6bbd3633c047984f67 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:49.944118 5192 raft_consensus.cc:2454] T 28e681a7ebc549789ec0a6f1a75c45d9 P ac17d3b553b74c9da39e0ab307addacd [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 6.
W20260929 14:12:49.955304 5197 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.896s user 0.003s sys 0.049s
W20260929 14:12:49.955116 5272 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.730s user 0.016s sys 0.036s
W20260929 14:12:49.939468 8780 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.825s user 0.002s sys 0.021s
I20260929 14:12:49.839612 8702 raft_consensus.cc:504] T e55a9c8ae48e44c59fee506e539b3f90 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:49.935242 5269 raft_consensus.cc:1261] T 3565847d030f4cc49b113969f438c801 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Refusing update from remote peer d886fb1eb0474d2d9f7410c4013c45e4: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260929 14:12:49.959589 8554 raft_consensus.cc:482] T f84af62485a4405f887bb8675221641e P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 675b77a0a88a4bf08134281fc0cd949e)
I20260929 14:12:49.912046 8746 raft_consensus.cc:504] T 0e4a096fb48c46dd95a1d74d34b6e670 P d886fb1eb0474d2d9f7410c4013c45e4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:49.986799 5344 raft_consensus.cc:1226] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Rejecting Update request from peer 675b77a0a88a4bf08134281fc0cd949e for earlier term 8. Current term is 9. Ops: []
I20260929 14:12:49.983114 5200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "929c62a6106e42d08a13689e62dce3b1" candidate_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "ac17d3b553b74c9da39e0ab307addacd" is_pre_election: true
I20260929 14:12:49.985611 5349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a38ce5b8e5024c0ab37787aaf2b357c0" candidate_uuid: "675b77a0a88a4bf08134281fc0cd949e" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" is_pre_election: true
I20260929 14:12:49.881856 8688 raft_consensus.cc:504] T 64181b973d25469da29a63928ede710f P c5ec116e39144881bdf1f2cd3e923cf5 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "675b77a0a88a4bf08134281fc0cd949e" member_type: VOTER last_known_addr { host: "127.3.242.68" port: 35497 } attrs { promote: false } }
I20260929 14:12:49.954325 8337 raft_consensus.cc:504] T e61c0e0d757e43f19868f329ee0b8bec P d886fb1eb0474d2d9f7410c4013c45e4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:49.996256 8776 raft_consensus.cc:482] T a3673029b370458db49bab90bbd0bd82 P d886fb1eb0474d2d9f7410c4013c45e4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:49.957876 8589 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P c5ec116e39144881bdf1f2cd3e923cf5 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:49.954021 8807 raft_consensus.cc:482] T 4b277670ba5344b493c046b2a0864597 P ac17d3b553b74c9da39e0ab307addacd [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 14:12:50.005621 5272 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:50776 (request call id 2303) took 1929 ms. Trace:
I20260929 14:12:50.027904 5349 raft_consensus.cc:2454] T a38ce5b8e5024c0ab37787aaf2b357c0 P d886fb1eb0474d2d9f7410c4013c45e4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 675b77a0a88a4bf08134281fc0cd949e in term 12.
I20260929 14:12:50.028831 5200 raft_consensus.cc:2454] T 929c62a6106e42d08a13689e62dce3b1 P ac17d3b553b74c9da39e0ab307addacd [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d886fb1eb0474d2d9f7410c4013c45e4 in term 7.
I20260929 14:12:49.918808 8799 raft_consensus.cc:504] T 465f57e1a1654ed1b40ce88c27fa52ac P d886fb1eb0474d2d9f7410c4013c45e4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
I20260929 14:12:49.968037 8720 leader_election.cc:288] T 632bdccb08444a4d89ecc1cfdbab98b2 P d886fb1eb0474d2d9f7410c4013c45e4 [CANDIDATE]: Term 9 election: Requested vote from peers c5ec116e39144881bdf1f2cd3e923cf5 (127.3.242.66:43473), 675b77a0a88a4bf08134281fc0cd949e (127.3.242.68:35497)
I20260929 14:12:49.965767 8792 raft_consensus.cc:482] T 321633daf3914178a03ee03e5c0159d9 P ac17d3b553b74c9da39e0ab307addacd [term 8 FOLLOWER]: Starting pre-election (detected failure of leader d886fb1eb0474d2d9f7410c4013c45e4)
I20260929 14:12:50.030404 8754 raft_consensus.cc:2726] T c15d9fe0c3724bfbad2156748acd468e P ac17d3b553b74c9da39e0ab307addacd [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260929 14:12:50.031069 5347 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:47252 (request call id 2500) took 4195 ms (client timeout 2201 ms). Trace:
I20260929 14:12:50.061167 5272 rpcz_store.cc:276] 0929 14:12:48.071855 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:48.071985 (+ 130us) service_pool.cc:227] Handling call
0929 14:12:48.072313 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0929 14:12:48.072504 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 5
0929 14:12:48.224784 (+152280us) log.cc:842] Serialized 14 byte log entry
0929 14:12:50.000852 (+1776068us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0929 14:12:50.000919 (+ 67us) raft_consensus.cc:1631] Marking committed up to 5
0929 14:12:50.000979 (+ 60us) raft_consensus.cc:1683] Filling consensus response to leader.
0929 14:12:50.001058 (+ 79us) raft_consensus.cc:1677] UpdateReplicas() finished
0929 14:12:50.001203 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":6579,"prepare.run_cpu_time_us":22,"prepare.run_wall_time_us":23,"thread_start_us":1926532,"threads_started":2}
I20260929 14:12:50.060045 8538 raft_consensus.cc:482] T ae31abec82ff4f40b695ccb19f289dbc P 675b77a0a88a4bf08134281fc0cd949e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
I20260929 14:12:50.048950 8801 raft_consensus.cc:482] T b7924b2689dc46c98af8c4344a1e1e33 P 675b77a0a88a4bf08134281fc0cd949e [term 10 FOLLOWER]: Starting pre-election (detected failure of leader ac17d3b553b74c9da39e0ab307addacd)
W20260929 14:12:50.073350 8798 log.cc:924] Time spent T 0c3538be0f624dd79d0e667313b22d02 P c5ec116e39144881bdf1f2cd3e923cf5: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20260929 14:12:50.047485 8496 consensus_queue.cc:237] T f7507b3dbd59424f8887991236a4a49c P ac17d3b553b74c9da39e0ab307addacd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac17d3b553b74c9da39e0ab307addacd" member_type: VOTER last_known_addr { host: "127.3.242.65" port: 45147 } } peers { permanent_uuid: "c5ec116e39144881bdf1f2cd3e923cf5" member_type: VOTER last_known_addr { host: "127.3.242.66" port: 43473 } } peers { permanent_uuid: "d886fb1eb0474d2d9f7410c4013c45e4" member_type: VOTER last_known_addr { host: "127.3.242.67" port: 42639 } }
W20260929 14:12:50.046008 8753 consensus_meta.cc:287] T 0c3538be0f624dd79d0e667313b22d02 P ac17d3b553b74c9da39e0ab307addacd: Time spent flushing consensus metadata: real 1.366s user 0.006s sys 0.005s
I20260929 14:12:50.103713 5870 leader_election.cc:302] T a38ce5b8e5024c0ab37787aaf2b357c0 P 675b77a0a88a4bf08134281fc0cd949e [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 675b77a0a88a4bf08134281fc0cd949e, d886fb1eb0474d2d9f7410c4013c45e4; no voters:
W20260929 14:12:50.080003 5347 rpcz_store.cc:269] 0929 14:12:45.808943 (+ 0us) service_pool.cc:170] Inserting onto call queue
0929 14:12:45.809360 (+ 417us) service_pool.cc:227] Handling call
0929 14:12:50.004431 (+4195071us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54002599d4. stack: 00007fbda1b13033 00007fbda1b12f30 00007fbda1471534 00007fbdab17d8b4 00007fbdab17d849 00007fbda49f2471 00007fbda4a712ae 00007fbda99aec27 00007fbda3096c04 00007fbda3096b92 00007fbda3096ac1 00007fbda3096a3c 00007fbda3095cf2 00007fbda2e9794c 00007fbda2e96d86 00007fbda2e9672f
0929 14:12:50.004581 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles"